2014/05/22
22 May, 2014

High Availability Deployment in WSO2 Complex Event Processor

  • Shashika Ubhayaratne
  • Associate Technical Lead - QA - WSO2
Archived Content
This article is provided for historical perspective only, and may not reflect current conditions. Please refer to relevant product page for more up-to-date product information and resources.

Introduction

  • The high availability feature
  • Deployment scenarios
  • Configuring the CEP to achieve “high availability”
  • Sample
    • Configuring the ESB
    • Configuring the CEP
  • Conclusion

Introduction

WSO2 CEP is a lightweight, easy-to-use, purely open source product. It identifies most meaningful patterns, relationships, and data abstractions among a set of unrelated events and triggers immediate responses real time.

The CEP has an extremely high performance processing engine that can process millions of events per second. With the usage of Siddhi Query Language (SiddhiQL), the CEP can process streams and identify complex event occurrences in a highly scaled manner. It can support large numbers of queries via different Siddhi level features and hazelcast support.


The high availability feature

The CEP introduced the "high availability" feature in the newest release (version 3.1). This is one of the distributed deployment patterns that allows two CEP nodes to work on a cluster in “active” mode where both nodes process all the data. However, only one node sends out the notifications: if the notifying node fails, the other node will start sending notifications. This article will discuss a sample scenario using WSO2 ESB’s BAM server profile and CEP.

For this you may need to download ESB and CEP binaries from the following locations:

https://wso2.com/products/enterprise-service-bus/

https://wso2.com/products/complex-event-processor/


Deployment scenarios

When considering deployment patterns of "high availability", there are three types that will be discussed below

  1. Full active-active mode:

    Clients can send requests to both nodes if they have the same event stream and use the stream in the distributed execution plan. For this purpose, both execution plans should have the same name. This is depicted in Figure 1.

    Figure 1

  2. Active-active mode:

    Clients can send requests between two CEP nodes in a round-robin manner.

  3. Failover mode:

    Clients send all the requests to one node and it becomes an “active” node. When the active node fails, the requests are sent to the passive node. The state of the event is also maintained in the passive node. Since the failover mode provides better guarantee of events ordering over the active-active mode, it is recommended to use the pattern illustrated in Figure 2.

    Figure 2

    In the second and third patterns, the CEP is configured to send events to other nodes.


Configuring the CEP to achieve “high availability”

Two CEP servers are configured as management nodes. Fronting them with a load balancer is not obligatory.

  1. Enable clustering in both nodes as a “wka” (well-known address)

    This needs to be configured in axis2.xml (CEP_HOME/repository/conf/axis2/axis2.xml)

      
    
    wka
    
  2. 2. Define both nodes as members
      
    
                
                    127.0.0.1
                    4000
                
                  
                    127.0.0.1
                    4001
                
            
    
    Node1 axis2.xml:
    
        ...
        
            ...    	
            wka
            ...
            4000
            ...
            
                
                    127.0.0.1
                    4000
                
                  
                    127.0.0.1
                    4001
                
            
        ...
        
    
    Node2 axis2.xml:
    
        ...
        
            ...    	
            wka
            ...
            4001
            ...
            
                
                    127.0.0.1
                    4000
                
                  
                    127.0.0.1
                    4001
                
            
        ...
        
    
    

    Note: Correct IP address or hostname needs to be provided

  3. Set Node 2 to start with an offset if both nodes are running on the same machine (CEP_HOME/repository/conf/carbon.xml)
      
    
    
        ...
        
            1
        
        ...
    
    
  4. Share the registry databases

    Here we use the H2 database of CEP Node1 with Node2 (for H2 database make AUTO_SERVER=TRUE)

    Node1: master-datasource.xml (CEP_HOME/repository/conf/datasources/master-datasources.xml

      
    
        ...
        
            ...
            
                WSO2_CARBON_DB
                The datasource used for registry and user manager
                
                    jdbc/WSO2CarbonDB
                
                
                    
                        jdbc:h2:repository/database/WSO2CARBON_DB;AUTO_SERVER=TRUE
                        wso2carbon
                        wso2carbon
                        org.h2.Driver
                        50
                        60000
                        true
                        SELECT 1
                        30000
                    
                
            
            ...
        
        ...
    
    
    Node2: master-datasource.xml
    
        ...
        
            ...
            
                WSO2_CARBON_DB
                The datasource used for registry and user manager
                
                    jdbc/WSO2CarbonDB
                
                
                    
                        jdbc:h2:/repository/database/WSO2CARBON_DB;AUTO_SERVER=TRUE
                        wso2carbon
                        wso2carbon
                        org.h2.Driver
                        50
                        60000
                        true
                        SELECT 1
                        30000
                    
                
            
            ...
        
        ...
    
    
  5. Create the execution plan as “RedundantNode” for distributed processing

    Note: When two servers are started, member joined messages can be seen on the second node terminal. For example, [2014-05-04 22:57:25,781] INFO - {WKABasedMembershipScheme} Member joined [807ecd12-e6b6-4a44-9b64-fef73ac320cf]: /127.0.0.1:4000


Sample

Configuring the ESB

Configuring BAM server profile from ESB (ESB 4.8.1)

  1. Start ESB and Navigate to “Configure → BAM Server Profile”
  2. Enter any name for the “Profile Name”
  3. User Name/ Password: admin/ admin (if you run using super admin. Otherwise correct username/ password or tenant login details needs to be given)
  4. Tick the check-box “Enable Load Balancer”
  5. The URL set should be given as follows: e.g.: tcp://localhost:7611|tcp://localhost:7612 (Correct thrift ports of both the CEP servers should be specified; e.g.: when two CEP servers run on default offset and offset 1: the correct thrift ports are 7611 and 7612)
  6. Create an ESB proxy with a BAM mediator for inSequence by selecting the above created BAM server profile details (Server Profile, Stream Name, Stream Version) and send the mediator for outSequence.
  


   
      
         
         
            
               
            
         
      
      
         
      
   
   


Configuring the CEP

Start both CEP servers and send the following cURL request to generate an event stream (ESB server is running on offset 2 and the proxy service name is “BAMProxy) curl -v https://shashi-ThinkPad-T530:8282/services/BAMproxy

Note: when the request is sent twice, both servers terminals will display the following line as the event stream is generated. INFO - {EventJunction} Producer added to the junction. Stream:esbStream1:1.0.0

Configure Node1 CEP:

  1. Create “ExecutionPlan” using the below configurations and steps;

    Execution Plan Name: ExecutionPlan

    Distributed processing:Disabled

    Import Stream: esbStream1:1.0.0

    As: inESBStream1

  2. Click “Import”
  3. Export Stream Value of: esbStream_Distributed
  4. StreamId: Select “Create Stream Definition” → “Create” → “Create Later” for event formatter window → “Add”
  5. Select “Add Execution Plan”
  6. Create “RedundantExecutionPlan” with below configurations

    Execution Plan Name: RedundantExecutionPlan

    Distributed processing:RedundantNode

    Import Stream: esbStream_Distributed:1.0.0

    As: outStream1

  7. Click “Import”
  8. Export Stream Value of: outStream2
  9. StreamId : Select “Create Stream Definition” → “Create” → “Create Default Logger Formatter” for event formatter window → “Add”
  10. Add an output WSO2 event adapter for Node2 (With Node 2 thrift port for Receiver URL*)
  11. Navigate to ‘Event Streams’ and click “Out Flows” in front of “esbStream1:1.0.0”
  12. Click on the link “Publish to External Event Stream (via Event Formatter)”
  13. Select “Output Event Adapter Name” as the adapter name created in Step 10
  14. Enter any name for the event formatter and stream
  15. Click “Add Event Formatter”
  16. The same steps need to be followed for node2 as well. Streams and the execution plan both need to be same
  17. Refer to the attached configuration for more information
  18. Send “curl -v https://shashi-ThinkPad-T530:8282/services/BAMproxy” for ESB proxy repeatedly
  19. The CEP terminal (active node) will display following event occurrences
      
    meta_http_method:GET,
    meta_character_set_encoding:UTF-8,
    meta_remote_address:127.0.0.1,
    meta_transport_in_url:/services/BAMproxy,
    meta_message_type:application/xml,
    meta_remote_host:127.0.0.1,
    meta_service_prefix:https://127.0.1.1:8282,
    meta_host:shashi-ThinkPad-T530.local:9765,
    correlation_activity_id:77361835918923663539,
    message_direction:IN,
    service_name:BAMproxy,
    operation_name:mediate,
    message_id:urn:uuid:eb44d987-d3cc-42d4-b7e0-b9ab496f891f,
    timestamp:1399225969282
    [2014-05-04 23:22:49,330]  INFO - {ReceiverGroup}  Resending the failed published data…
    

Failover scenario verification

  1. Shut down the active node (node which receives the events)

    Note: The passive CEP node which maintained the state of the event internally becomes active and starts to receive events.

  2. Verify the other CEP node’s terminal
      
    2014-05-07 11:25:04,210]  INFO - {LoggerEventAdaptorType}  Unique ID : event , Event : meta_tenant_id:-1234,
    meta_http_method:GET,
    meta_character_set_encoding:UTF-8,
    meta_remote_address:127.0.0.1,
    meta_transport_in_url:/services/BAMproxy,
    meta_message_type:application/xml,
    meta_remote_host:127.0.0.1,
    meta_service_prefix:https://127.0.1.1:8282,
    meta_host:shashi-ThinkPad-T530.local:9765,
    correlation_activity_id:10281016950496117197667,
    message_direction:IN,
    service_name:BAMproxy,
    operation_name:mediate,
    message_id:urn:uuid:706bd910-5405-406c-88e0-cf8c014238a8,
    timestamp:1399442104207
    

Conclusion

This article has explained the “high availability” feature, its different deployment patterns, and the most common and recommended sample scenarios. The CEP is very significant when it comes to real-time event processing and high availability is the most important feature that provides data availability and ensures that no data is lost; even when an active CEP node fails to function, a passive node becomes active and will process events.

 

About Author

  • Shashika Ubhayaratne
  • Associate Technical Lead - QA
  • WSO2