Table of contents

• Overview
• Starting servers
• Expose a WSO2 AS hosted web service through WSO2 ESB
• Publish important data to WSO2 BAM
• Write analytics on received data in BAM
• Visualize, analyzed in a BAM dashboard

Overview

This tutorial demonstrates how to

• Expose a WSO2 AS hosted web service through WSO2 ESB
• Publish important data to WSO2 BAM
• Write analytics on received data in BAM
• Visualize, analyzed in a BAM dashboard

Here, we use the operation, getSimpleQuote in webservice SimpleStockQuoteService, which is a readily available sample in WSO2 ESB. It is a service that shows stock information for the given company name. Finally, we are going to view the average stock volume viewed in each company.

Starting servers

Download the following servers from the given locations.

1. WSO2 AS
2. WSO2 ESB
3. WSO2 BAM

For this demonstration, we have used WSO2 AS 5.1.0, WSO2 ESB 4.6.0 and WSO2 BAM 2.3.0.

Extract each zip packed server, e.g. $ unzip wso2as-5.1.0.zip for extracting WSO2 Application Server in Linux environments.

Then, each server should be set in different ports as all the servers will be started in the same machine (i.e. same IP). Ports of each server can be changed by changing the Port Offset of each server in different numbers from the default value, zero. To change the port offset, change the port offset element in <Carbon_Home>/repository/conf/carbon.xml.

    <!-- Ports offset. This entry will set the value of the ports defined below to the define value + Offset.
     e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445
    -->
    <Offset>0</Offset>

For this exercise, set offset values as follows.

1. WSO2 AS - Offset = 2
2. WSO2 ESB - Offset = 1
3. WSO2 BAM - Do not change where Offset will be 0

Start all three servers.

Expose a WSO2 AS hosted web service through WSO2 ESB

First, go the WSO2 ESB home directory and build the service (you need Apache Ant to be installed in the computer earlier).

    $ cd <WSO2_ESB_Home>/samples/axis2Server/src/SimpleStockQuoteService
    $ ant

Then, deploy the SimpleStockQuoteService.aar located in <WSO2_ESB_Home>/samples/axis2Server/repository/services as given in this document.

If you have successfully deployed the web service, you will see it in UI as follows when you follow the path Main → Services → List , in the WSO2 AS Management Console UI.

You can test the service by clicking on Try this service in each service. The following will be the template of the request message of the SimpleStockQuoteService you would be able to see when you click that link. Here, we have selected the operation, getSimpleQuote from the top left corner menu as shown because we would be using this operation for the rest of this tutorial.

You can test this service by replacing the ? character in the xs:symbol element with the company name and by clicking on the Send button. The response will be shown in the Response column. As this is a sample service, it will generate some random values as results for each request.

In this tutorial, we extract the values from the XML elements, ax21:name and ax21:volume in the response message, which has values, ABC Company and 8107 in this example.

Now, add the deployed web service to a passthrough proxy service in ESB as described here.

Here, enter SimpleStockQuoteProxy as the Proxy Service Name.

Select Enter URL option as Target Endpoint. Then enter https://localhost:9765/services/SimpleStockQuoteService as the Target URL and create the proxy service.

Publish important data to WSO2 BAM

In WSO2 BAM, it is possible to publish ESB data to BAM in two main ways:

1. Publishing whole message header/body of each message
2. Publishing only the required data after extracting them from the message

The first instance will dump a lot of data to BAM at each message passing through the ESB, which will take a large disk space in the Cassandra database, and will require high network traffic if Cassandra is located in a different location; however, it will not increase the processing work that takes place in the ESB.

In contrast to the first way, in the second way, required values will be extracted from the message for the given XPath and will be casted to the required data types before it is sent to the BAM. This will increase the processing that takes place inside the ESB, but will reduce the data flow rate from ESB to BAM and will save the Cassandra disk space.

Due to the ability of explaining the idea in this article in a much simpler way, we use the second way to publish data to BAM from the ESB.

Here, we intercept the response message from the web service, SimpleStockQuoteService, and extract the name and volume tag values in the response message using Property mediators and set them as properties. Then, these two properties are captured in the BAM mediator and sent to the BAM server. The get-property method is used to capture the property in the BAM mediator as shown in the document (the example shows how to capture the property, SYSTEM_DATE. Similarly, these two properties can also be captured) name property is sent as a String type field and the volume property is casted as an integer by selecting the required type for each property as shown. Remember to untick the Dump Header and Dump body check boxes as they are not used in our analytics.

The following is the BAM mediator configuration BAM Server Profile created with the name SimpleStockQuoteProfile in ESB.

The next step is to add a BAM mediator into the created proxy service, SimpleStockQuoteProxy. Moreover, there should be two more property mediators included into the same proxy service before the BAM mediator. You can read the docs for the Property mediator if you want to know how to add a Property mediator to extract fields from an XML message. Read from the 7th step of the section, Adding a BAM Mediator to WSDL Proxy Service in BAM mediator document to know how to add a BAM mediator to the proxy service. Here, you can add the created BAM Server Profile, SimpleStockQuoteProfile, stream name, ssqs and version, 1.0.0 as shown. Remember that these three mediators should be added only to the Out Sequence of the proxy service before the Send mediator.

Shown below is the source view of the complete proxy service.

    <proxy xmlns="http://ws.apache.org/ns/synapse" name="SimpleStockQuoteProxy" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">
        <target>
            <outSequence>
                <property xmlns:ax21="https://services.samples/xsd" xmlns:ns="https://services.samples" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" name="SSQS_VOLUME" expression="//ns:getSimpleQuoteResponse/ns:return/ax21:volume" scope="default" type="STRING"/>
                <property xmlns:ax21="https://services.samples/xsd" xmlns:ns="https://services.samples" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" name="SSQS_NAME" expression="//ns:getSimpleQuoteResponse/ns:return/ax21:name" scope="default" type="STRING"/>
                <bam>
                    <serverProfile name="SimpleStockQuoteProfile">
                        <streamConfig name="ssqs" version="1.0.0"/>
                    </serverProfile>
                </bam>
                <send/>
            </outSequence>
            <endpoint>
                <address uri="https://localhost:9765/services/SimpleStockQuoteService"/>
            </endpoint>
        </target>
        <description></description>
    </proxy>

Now, we need to check the accuracy of the above-mentioned configurations. You can simply send some data to webservice via the proxy service and see whether the sent company names and their stock volumes are correctly logged in the Cassandra database.

For testing the proxy service, go to menus, Main → Services → List in the ESB Management Console and click on the Try this service link of the SimpleStockQuoteProxy proxy service. Then, use the same request message template used in the Application Server, which was shown earlier.

For the demonstration, we have sent

• 2 messages with the symbol A
• 3 messages with the symbol B
• and 2 messages with the symbol C

If you login to the BAM Management Console and go to the menus, Main → Explore Cluster and enter the following connection parameters and click on the Connect button, you will see the Cassandra Keyspaces and their column families in the UI.

    Connection Url : localhost:9160
    User Name : admin
    Password : admin

As we have given the Stream Name as ssqs in the BAM mediator, events from this mediator will be stored in the ssqs column family in the Keyspace, EVENT_KS. Click on the ssqs link to explore the content in the column family.

By clicking on View more, the link for each message will show the message content in detail. Moreover, note that the event number is the number next to the last double colon separation of each Row ID (e.g. 1369552705503::192.168.1.2::9443::5 is the Row ID of 5th message sent to this column family).  

The following is the 5th message content.

Write analytics on received data in BAM

At the moment, we could successfully publish the selected data events of the WSO2 AS response message to the WSO2 BAM. From this point onwards, we are going to process the Cassandra data using Hadoop map-reduce jobs. For this, we use the Hive language, which simplifies the map-reduce interface of Hadoop. Hive has a syntax that's similar to that of SQL. Read this doc to get started with Hive syntax, which are generally used in BAM analytics.

The following is the Hive script we use for processing the data saved in Cassandra.

    CREATE EXTERNAL TABLE IF NOT EXISTS SimpleStockQuoteService (messageID STRING, companyName STRING, stockVolume INT)
    STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
    WITH SERDEPROPERTIES (
        "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" ,
        "cassandra.cf.name" = "ssqs" ,
        "cassandra.columns.mapping" =
        ":key, payload_name, payload_volume" );

    CREATE EXTERNAL TABLE IF NOT EXISTS SimpleStockQuoteServiceSummary (companyName STRING, stockVolumeAverage DOUBLE)
    STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler'
    TBLPROPERTIES (
        'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE',
        'hive.jdbc.update.on.duplicate' = 'true' ,
        'hive.jdbc.primary.key.fields' = 'companyName' ,
        'hive.jdbc.table.create.query' =
        'CREATE TABLE SimpleStockQuoteServiceSummary (companyName VARCHAR(100) NOT NULL PRIMARY KEY, stockVolumeAverage DOUBLE)' );

    insert overwrite table SimpleStockQuoteServiceSummary
        select companyName, avg(stockVolume) AS stockVolumeAverage
        from SimpleStockQuoteService
        group by companyName;

This script basically,

1. Creates a Hive table SimpleStockQuoteService wrapping the column family, ssqs
2. Creates a Hive table SimpleStockQuoteServiceSummary, while creating a RDBMS table, SimpleStockQuoteServiceSummary, which is wrapped by the Hive table
3. Updates Hive table, SimpleStockQuoteServiceSummary with the Hive table, SimpleStockQuoteService averaging the stock volume for each company

You need to add this Hive script to BAM with the name, simple_stock_quote_service as shown. You can refer to this document for more details.

You can schedule this script for a specific time duration as described in this doc or manually execute by clicking on the Execute button. As this script is used to continuously view the requests invoked from the Application Server, scheduling is required. Time duration should be selected based on the maximum latency you expect and the processing power of Hadoop you are using.

Visualize, analyzed in a BAM dashboard

As you can see in the Hive script, the summarized data (i.e. averaged stock quote volumes) is stored in the SimpleStockQuoteServiceSummary RDBMS database. As H2 is the default packed database that comes with WSO2 BAM, there should be a table created in the H2 database with the name, SimpleStockQuoteServiceSummary.

Now, we are going to generate a BAM dashboard gadget according to this doc.

First go to the Tools → Gadget Gen Tool in BAM Management Console. Then enter the values given below, which are related to the data source, WSO2BAM_DATASOURCE mentioned in <BAM_Home>/repository/conf/datasources/master-datasources.xml (note how WSO2BAM_DATASOURCE is used in the Hive script given above).

    JDBC URL : jdbc:h2:repository/database/samples/BAM_STATS_DB;AUTO_SERVER=TRUE
    Driver Class Name : org.h2.Driver
    User Name : wso2carbon
    Password : wso2carbon

In the next step, add the following SQL script.

    SELECT * FROM SimpleStockQuoteServiceSummary

In the third step, enter as follows.

    Pick UI Element : Bar Graph
    Chart Title : Simple Stock Quote Volume Agerage
    Y-Axis Label : Average Volume
    Y-Axis Column : STOCKVOLUMEAVERAGE
    X-Axis Label : Company Name
    X-Axis Column : COMPANYNAME

In the 4th step, enter as follows. Note that the Refresh Rate also should be selected intelligently as the Hive script scheduling time duration.

    Gadget Title : Simple Stock Quote Value Average
    Gadget File Name : ssqva
    Refresh Rate (in Seconds) : 60

Then click on the Generate button and follow the steps as mentioned in the doc. You will get the final dashboard as follows.

As the SimpleStockQuoteService generates stock quote volumes that are randomly independent from the input symbol, there will not be a significant difference between each company’s stock quote averages. And if you take averages for many number of requests for each company name, averages will seem almost the same.

Author

Maninda Edirisooriya, Software Engineer, WSO2.