[Tutorial] Configuring highly available WSO2 BAM setup to analyze WSO2 API Manager usage statistics
- Suneth Ranasinghe
- Senior Software Engineer - QA - WSO2
Introduction
WSO2 API Manager 1.5.0 is now released and supports load balanced data publishing when integrated with WSO2 Business Activity Monitor (BAM) server 2.3.0. This tutorial will provide a guide on how to cluster the BAM server's two nodes and use that for publishing load balanced data in WSO2 API manager to monitor API usage and performance statistics even in a fail-over scenario, which will result in one WSO2 BAM server to stop working.
Contents
- Deployment architecture
- Prerequisites
- Sharing registry space across API Manager and multiple BAM instances
- Clustering Cassandra and BAM data receivers
- Configuring the analyzer engine
- Server startup
- Visualizing results
- Conclusion
1. Deployment architecture
2. Prerequisites
2.1. Downloading products
- Download WSO2 BAM 2.3.0 here.
- Download WSO2 API Manager 1.5.0 here.
- Download Zookeeper 3.3.6 here. (Only BAM 2.3.0 and lower versions, use the zookeeper for the task coordination.)
2.2. Installing the products
- Unzip WSO2 BAM 2.3.0 to two machines, i.e.; 10.100.1.1 and 10.100.1.2 to a folder of your choice. We will name these folders $BAM_HOME hereon.
- Unzip WSO2 API-M 1.5.0 to a preferred location. This location will be referred to as $APIM_HOME.
- Extract the downloaded Zookeeper pack to a location in BAM node 1 (10.100.1.1) and BAM node 2 (10.100.1.2). From now on, these locations are referred to as $ZOO_HOME.
2.3. Increasing heap memory and syncing of BAM nodes
- Increase heap memory of both BAM receivers to 2GB by updating the wso2server.sh file of each node. Default setting for WSO2 API Manager 1.5.0 : -Xms256m -Xmx1024m -XX:MaxPermSize=256m. New settings : -Xms2048m -Xmx2048m -XX:MaxPermSize=1024m
- Sync the time in all BAM nodes.
2.4. Creating MySQL databases
Create the following databases in MySQL.
CREATE DATABASE bam_carbondb;
USE bam_carbondb;
SOURCE {CARBON_HOME}/dbscripts/mysql.sql
GRANT ALL ON bam_carbondb.* TO bam_carbondb@'%' IDENTIFIED BY "bam_carbondb";
CREATE DATABASE bam_regdb;
USE bam_regdb;
SOURCE {CARBON_HOME}/dbscripts/mysql.sql
GRANT ALL ON bam_regdb.* TO bam_regdb@'%' IDENTIFIED BY "bam_regdb";
CREATE DATABASE bam_datadb; GRANT ALL ON bam_datadb.* TO bam_datadb@'%' IDENTIFIED BY "bam_datadb";
2.5. Calculating tokens for data centers
Follow the instructions below to generate tokens for the data centers available in the cluster. The steps are given below.- Create a new file for your token generator program as follows;
- Paste the following Python program into this file;
- Save and close the file and make it executable:
- Run the script:
- When prompted, enter the total number of nodes in your cluster:
vi tokengentool
#! /usr/bin/python
import sys
if (len(sys.argv) > 1):
num=int(sys.argv[1])
else:
num=int(raw_input("How many nodes are in your cluster? "))
for i in range(0, num):
print 'token %d: %d' % (i, (i*(2**127)/num))
chmod +x tokengentool
./tokengentool
token 0: 0 token 1: 56713727820156410577229101238628035242
3. Sharing registry space across API Manager and Multiple BAM instances
This section will explain how API Manager is enabled for track API usage and how the API Manager registry is shared across the two BAM nodes3.1. Enabling API Manager for tracking API usage
- Configure the
in the /repository/conf/api-manager.xml as follows;
<APIUsageTracking>
<!-- Enable/Disable the API usage tracker. -->
<Enabled>true</Enabled>
<!-- API Usage Data Publisher. -->
<PublisherClass>org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher</PublisherClass>
<!-- Thrift port of the remote BAM server. ->
<ThriftPort>7611</ThriftPort>
<!-- Server URL of the remote BAM/CEP server used to collect statistics. Must
be specified in protocol://hostname:port/ format. -->
<BAMServerURL>tcp://10.100.1.1:7611, tcp://10.100.1.2:7611</BAMServerURL>
<!-- Administrator username to login to the remote BAM server. -->
<BAMUsername>admin</BAMUsername>
<!-- Administrator password to login to the remote BAM server. -->
<BAMPassword>admin</BAMPassword>
<!-- JNDI name of the data source to be used for getting BAM statistics.
This data source should be defined in the master-datasources.xml file in conf/datasources directory. -->
<DataSourceName>jdbc/WSO2AM_STATS_DB </DataSourceName>
</APIUsageTracking>
Make sure to mention both BAM server URLs in
3.2. Configuring API Manager for mounting
1. Go to $APIM_HOME/repository/conf/datasources and edit the WSO2_REG_DB of the master-datasources.xml to connect the API Manager instance to a MySQL database. Once edited, the configuration should be as follows.
<datasource>
<name> WSO2_REG_DB</name>
<description>The data source used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2RegDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://10.100.1.1:3306/bam_regdb?autoReconnect=true</url>
<username>bam_regdb</username>
<password>bam_regdb</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
2. Specify the data source definition in
<datasource>
<name>WSO2AM_STATS_DB</name>
<description>The datasource used for getting statistics to API Manager</description>
<jndiConfig>
<name>jdbc/WSO2AM_STATS_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<!-- JDBC URL to query the database -->
<url>jdbc:mysql://10.100.1.1:3306/bam_datadb?autoReconnect=tru</url>
<username>bam_datadb</username>
<password>bam_datadb</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
3. Drop the mysql-connector-java-5.1.x-bin.jar to $APIM_HOME/repository/components/lib folder.
4. Set the ports Offset to 1000 (since we need BAM to run with default ports, we are increasing the offset in API Manager) by editing the file
<Offset>1000</Offset>
3.3. Configuring BAM nodes for mounting
1. Go to $BAM_HOME/repository/conf/datasources of BAMNode1 and edit the WSO2_CARBON_DB of the master-datasources.xml to connect the local registry space of BAM instance to an MySQL database. Once edited, the configuration should be as follows.
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The data source used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://10.100.1.1:3306/bam_carbondb?autoReconnect=true</url>
<username>bam_carbondb</username>
<password>bam_carbondb</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
2. Add the below data source configuration to the same file (master-datasource.xml). We will be pointing to this database through the user-mgt.xml as to store User permissions.
<datasource>
<name> WSO2_USER_DB</name>
:
<name>jdbc/WSO2UserDB</name>
:
<url>jdbc:mysql://10.100.1.1:3306/bam_regdb?autoReconnect=true</url>
<username>bam_regdb</username>
<password>bam_regdb</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
:
</datasource>
3. The below data source configuration is used to connect to the MySQL database, which will be used to store analyzed data while the Hive script executes.
<datasource>
<name>WSO2AM_STATS_DB</name>
<description>The datasource used for analyzer data</description>
<jndiConfig>
<name>jdbc/WSO2AM_STATS_DB</name>
</jndiConfig>
:
<url>jdbc:mysql://10.100.1.1:3306/bam_datadb?autoReconnect=true</url>
<username>bam_datadb</username>
<password>bam_datadb</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
:
</datasource>
4. Add the following datasource configuration to the same master-datasources.xml file.
<datasource>
<name>WSO2BAM_CASSANDRA_DATASOURCE</name>
:
<url>jdbc:cassandra://10.100.1.1:9160/EVENT_KS</url>
<username>admin</username>
<password>admin</password>
:
</datasource>
5. Open the $BAM_HOME/repository/conf/registry.xml and add the below mounting configuration.
<dbConfig name="wso2Config">
<dataSource>jdbc/WSO2UserDB</dataSource>
</dbConfig>
<remoteInstance url="https://10.100.1.2:10443/registry">
<id>instanceId</id>
<dbConfig>wso2Config</dbConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<enableCache>true</enableCache>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceId</instanceId>
<targetPath>/_system/config</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceId</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
6. Drop the mysql-connector-java-5.1.x-bin.jar to $APIM_HOME/repository/components/lib folder
7. Edit the $BAM_HOME/repository/conf/user-mgt.xml file and change the jndi source name as below.
<Property name="dataSource">jdbc/WSO2UserDB</Property>
8. Except Step 3, follow the the above steps for BAM node 2 as well.
9. For BAM Node 2, when configuring Cassandra datasource, add the following datasource configuration to the master-datasources.xml file
<datasource>
<name>WSO2BAM_CASSANDRA_DATASOURCE</name>
:
<url>jdbc:cassandra://10.100.1.2:9160/EVENT_KS</url>
<username>admin</username>
<password>admin</password>
:
</datasource>
4. Clustering Cassandra and BAM data receivers
1. Open the cassandra.yaml file that resides in $BAM_HOME/repository/conf/etc directory. Give the cluster name in the file as WSO2BAM Cluster. -> cluster_name:'WSO2BAMCluster' Note: Execute the above step for both BAM nodes.
2. Edit the Cassandra configuration for BAM N2 (10.100.1.2) in $BAM_HOME/repository/conf/ etc/cassandra.yaml to be as follows.
cluster_name:WSO2BAMCluster
initial_token: 56713727820156410577229101238628035242
seed_provider:
-seeds:"10.100.1.1"
listen_address:10.100.1.2
rpc_address: 10.100.1.2
rpc_port: 9160
3. Cassandra configuration for BAM node 1 (10.100.1.1) in $BAM_HOME/repository/conf/etc/cassandra.yaml should be as follows.
cluster_name:WSO2BAMCluster initial_token:0 seed_provider: -seeds:"10.100.1.1" listen_address:10.100.1.1 rpc_address: 10.100.1.1 rpc_port: 9160
4. Edit the cassandra-component.xml file residing in $BAM_HOME/repository/conf/etc. Make sure this file contains the cluster name as follows.
cluster_name:WSO2BAMClusterNOTE: Follow the above step for both BAM nodes.
5. Change the cassandra-component.xml configuration as follows of both BAM nodes;
<Cassandra>
<Cluster>
<Name>WSO2BAMCluster</Name>
<Nodes>10.100.1.1:9160,10.100.1.2:9160</Nodes>
<DefaultPort>9160</DefaultPort>
<AutoDiscoverydisable="false"delay="1000"/>
</Cluster>
</Cassandra>
6. In all BAM nodes (nodes 1 and 2) edit the $BAM_HOME/repository/conf/advanced/streamdefn.xml as follows in order to change the replication factor and read/write consistency levels.
<StreamDefinition>
<ReplicationFactor>2</ReplicationFactor>
<ReadConsistencyLevel>ALL</ReadConsistencyLevel>
<WriteConsistencyLevel>ALL</WriteConsistencyLevel>
<StrategyClass>org.apache.cassandra.locator.SimpleStrategy</StrategyClass>
</StreamDefinition>
7. In BAM node 1 (10.100.1.1) add a file named cassandra-endpoint.xml in $BAM_HOME/repository/conf/etc with following configuration.
<Cassandra>
<EndPoints>
<EndPoint><HostName>10.100.1.1</HostName></EndPoint>
<EndPoint><HostName>10.100.1.2</HostName></EndPoint>
</EndPoints>
</Cassandra>
8. In BAM node 1 modify cassandra-auth.xml in $BAM_HOME/repository/conf/etc as follows.
<Cassandra>
<!-- localtransport-->
<!--<EPR>local://services/CassandraSharedKeyPublisher</EPR>-->
<!--HTTPtransport-->
<EPR>https://10.100.1.1:9443/services/CassandraSharedKeyPublisher</EPR>
<User>admin</User>
<Password>admin</Password>
</Cassandra>
Change to the HTTPS transport and give the BAM node 1 IP as highlighted above.
9. In BAM node 1 modify the cache.xml inside $BAM_HOME/repository/conf/etc as follows.
<cache xmlns="https://wso2.org/ns/2010/09/caching">
<cacheManager>org.wso2.carbon.caching.infinispan.InfinispanCacheManager</cacheManager>
<maxExpirationMillis>1800000</maxExpirationMillis>
<maxIdleExpirationMillis>1800000</maxIdleExpirationMillis>
</cache>
5. Configuring the analyzer engine
5.1. Clustering analyzer node
With configured and governance registries shared, now we can configure analyzer clustering between BAM node 1 and node 2.1. Uninstall the BAM Tool Box Deployer feature in BAM node 1 since having deployers in both BAM nodes interferes with the proper Hive task fail-over functionality. We will be using only BAM node 2 for tool box deployment.
- Start the analyzer node (10.100.1.1)
- Go to Home > Configure > Features
- Click in Installed Features tab
- Select the feature BAM Tool Box Deployer, which is already in the installed feature list
- Click on Uninstall
- Shutdown the server
2. Also disable registry caching in registry.xml in analyzer node (10.100.1.1) since it might cause intermittent issues in task scheduling.
<remoteInstance url="https://localhost:9443/registry">
<id>instanceId</id>
<dbConfig>wso2Config</dbConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<enableCache>false</enableCache>
</remoteInstance>
5.2. Configuring ZooKeeper
1. Create a file named ‘zoo.cfg’ inside $ZOO_HOME/conf with following content under BAM node 1.
tickTime=2000 dataDir=$ZOO_HOME/data clientPort=2181 tickTime=2000 initLimit=5 syncLimit=2 server.1=10.100.1.1:2888:3888 server.2=10.100.1.2:2888:3888Note: Replace $ZOO_HOME with the absolute path of the zookeeper distribution E.g.:- dataDir=C:\\BAM_Installation\\zookeeper-3.3.6\\data Also ensure you use the correct IPs of the two BAM nodes in the above configuration.
2. Create a new directory called ‘data’ under $ZOO_HOME to hold Zookeeper data (In both Node1 & Node 2).
3. Add a file with name ‘myid’ containing server number in ‘data’ directory which you created above.
E.g: For BAM node 1, create the myid file with 1 as its content. For BAM node 2, the myid file should have 2 as its content4. Follow Steps 1 to 3 for BAM node 2 as well.
5. Go to $BAM_HOME/repository/conf/etc directory in BAM node 1 and BAM node 2 and edit the ‘coordination-client-config.xml’ as follows:
<CoordinationClientConfigurationenabled="true">
<Servers>
<Serverhost="10.100.1.1"port="2181"/>
<Serverhost="10.100.1.2"port="2181"/>
</Servers>
<SessionTimeout>5000</SessionTimeout>
</CoordinationClientConfiguration>
Note: Do this configuration change in both BAM nodes
6. Go to $BAM_HOME/repository/conf/etc directory in the both BAM nodes and edit the ‘tasks-config.xml’ as follows:
<taskServerMode>CLUSTERED</taskServerMode>
<taskServerCount>1</taskServerCount>
6. Server startup
1. Start Zookeeper on BAM node 1 as follows.
Go to $ZOO_HOME/bin chmod u+x zkServer.sh ./zkServer.sh start (To stop ./zkServer.sh stop)2. Start the Zookeeper instance in BAM node 2 using the same commands as above
3. Then start the analyzer node using the command as specified below.
E.g.:- ./wso2server.sh -Dsetup4. Start the Presentation node (Node 2 - 10.100.1.2).
5. Install the toolbox to Node 2 (10.100.1.2)
NOTE: Startup all servers with the following command from BAM_HOME/bin. $ nohup sh wso2server.sh 2>&16. Start WSO2 API Manager server with the parameter -Dsetup to create the required tables.
E.g.:- ./wso2server.sh -Dsetup7. Visualizing results
The next step is to visualize the API statistics through the publisher. Once you click on the statistics link on the publisher, you will see the following as shown in the below image;
In a scenario such as when one BAM server stops running, the other BAM server would be able to collect all the data present in the API statistics, continuously, e.g. you can simply stop the BAM server running on node 1 (10.100.0.2) using ./wso2server.sh stop command and check if the publisher API statistics link presents the same analyzed results as before.
8. Conclusion
A successfully configured WSO2 BAM cluster, which is capable of rendering and producing statistics of APIs hosted and managed in the API Manager, would presents various statistical dashboards in the API publisher, depending on user granted permission levels.