WSO2 CEP in Action - An Analysis of Use in Real-World Applications of Different Domains
By Sajith Ravindra
- 5 Aug, 2014
Most enterprise transactions and activities consist of constant streams of events. An organization might have access to a large number of such event streams resulting in millions of events per second. These event streams might be concealing very valuable information that is not apparent.
In order to derive valuable information from these event streams, they have to be analyzed using different strategies in real time. Then, actions have to be taken based on the analysis in a timely manner to gain maximum advantage over competitors.
The WSO2 Complex Event Processor (CEP) can listen to streams of events generated by different sources in different formats and analyze them in real time, thereby enabling you to proactively take effective action in response to specific scenarios.
This article presents scenarios where we used the WSO2 CEP to solve problems in different domains that represent real-world applications. This article does not intend to give technical details of the implementation, and instead focuses on the application of WSO2 CEP for different use cases.
Analyzing a game of football
ACM International Conference on Distributed Event-Based Systems (DEBS) is one of the major conferences dedicated to event based computing. For this conference they provide a challenge to be solved using an event based system. The WSO2 CEP was used to successfully solve this challenge.
In 2013, the challenge was to analyze a game of football using an event-based system. To solve the problems provided in this challenge, WSO2 CEP had to process streams of events generated by the sensors attached to players and the ball.
Each sensor emits events at a rate of 60Hz. An event consists of the sensorID, timestamp, x,y,z coordinates, velocity and acceleration vectors. By using this data, the following information can be derived;
- It measures each player’s running speeds and calculates how long he spent on different speed ranges
- It calculates the duration each player kept the ball in their possession throughout the match
- It detect hits on the ball and detects goals
- If the ground is divided into a grid, the duration each player has spent in a given cell can be derived
This information was derived using WSO2 CEP with a set of Siddhi queries. In addition to the ones shown above, more information is required such as
- successful versus. failed passes
- shots on target
- offside detection
While performing a comprehensive analysis on the events, WSO2 CEP was able to produce a throughput of 100,000+ events.
In addition to real-time complex event processing, WSO2 CEP supports dashboards with a comprehensive set of gadgets to produce rich representation of information in a user-friendly manner. The following is the dashboard presenting real-time information about the football match
Watch the Realtime Analytics for Football video for a screencast to see how the visualization of the game on 2D is running in sync with the actual game video playing side by side.
Solving DEBS 2013 Grand Challenge with WSO2 CEP/Siddhi - Detailed implementation details
Analyzing a Soccer Game with WSO2 CEP - Presentation on this scenario done at WSO2Con EU 2014
Analysis of energy consumption measurements in smart grids
This use case is based on the ACM International Conference on Distributed Event-Based Systems (DEBS) challenge, 2014. The data for the challenge is based on recordings that are originated from smart plugs, which have been deployed in private households.
Only four solutions were accepted out of about 25 submissions at the conference. The solution implemented using WSO2 CEP was accepted along with the solutions provided by Dresden University of Technology (Germany), Imperial College London, and Fraunhofer Institute (Germany).
These smart plugs are embedded with two sensors that measure different power consumption related metrics. An event consists of the sensor ID, timestamp, reading value, property indicating type of reading, smart plug ID, household ID, and house ID. This data is emitted once per second.
By analyzing these events, the following analytics had to be carried out:
- Short-term load forecasting based on the current load measurements
- Producing load statistics for real-time demand management
The challenge required competitors to provide both a single node solution and a distributed solution for each query. WSO2 CEP was successfully used to implement solutions for both cases with promising performance and accuracy.
The results gained by this challenge proved the high performance nature of WSO2 CEP and its ability to handle large volumes of data seamlessly.
The task of forecasting the load resulted in throughput of 400,000 events in a single node with 1s latency and nearly 1 million events throughput in a distributed setup.
The task of calculating load statistics involved finding the outliers as far as the energy consumption levels are concerned by calculating the median of each plug. This task was performed with a throughput of 1 to 3 million events and with a latency of less than 100ms in a single node.
- ACM DEBS Grand Challenge 2014: Smart grids, 4 Billion events, throughout in range of 100Ks - More details on how WSO2 CEP was used to solve the problem
- Solving DEBS Grand Challenge with WSO2 CEP - Presentation slides presented at the conference
- Solving the grand challenge using an open source CEP engine - Research paper on the results
- DEBS grand challenge - Detailed explanation of the challenge
The CEP based email monitor is a solution that provides functionality to carry out advanced monitoring and searching of emails. In this solution, WSO2 CEP was used in collaboration with WSO2 Enterprise Service Bus (ESB). The ESB was responsible for providing access to emails while the monitoring and and searching logic was implemented using the CEP server.
This solution was mainly implemented based on gmail. In addition to supporting gmail query language (for simple querying based on content, sender, recipients, etc.) the use of WSO2 CEP enables users to query incoming emails based on advanced criteria, such as frequency of a thread receiving emails, frequency of a label receiving emails, and received emails that have no response within a given period of time.
This email monitoring solution provides an easy-to-use language that is specifically designed for emails. This is simpler to implement and use instead of Siddhi Query Language (SQL).
The ESB accesses the email account of a given user and the emails in their inbox are sent to the CEP as a stream of events. Then, the user-written queries are executed against the incoming event stream. When a match take place, the CEP sends a response back to the ESB with a label or a tag and then the ESB will apply the tag/label to the matching email/mail thread.
The following are some simple sample use cases that can be implemented using the CEP based email monitor, which cannot be done using gmail query language. If someone wants to monitor the most active mail threads of a given day, he/she might write a query as shown below. It tags mail threads with the label “IMPORTANT” if there are more than 50 responses within a day
frequency > 50/d
add label IMPORTANT
The criteria for being IMPORTANT can be further fine grained by adding more conditions. The query below is an example where any thread that gets more than 50 responses in a day and is responded to by “firstname.lastname@example.org” or “email@example.com” are considered as important.
frequency > 50/d
add label IMPORTANT
Moreover, with this CEP based email monitor the user can configure the CEP to send notifications when a certain condition is met. The following query sends notifications to firstname.lastname@example.org if there are more than 100 mail threads for the label marketing.
label : (marketing)
and thread frequency > 100/d
send 'mail (to:email@example.com subject:important body:freq>100/d)
Fleet management using GPS tracking
In this use case, WSO2 CEP consumes streams of events that contain GPS coordinates of tracking devices. These events are processed through the CEP to derive information required for an effective fleet management solution.
In this solution, functionalities such as overspeed alarms, idle time alarms, and collision detection were implemented. The events are received in GEOJson format.
While the processing was done using the CEP server, a separate dashboard of the fleet management system was implemented using the Jaggery framework. Information derived using the CEP is presented in this dashboard.
The high speed device locations are shown through red color pointers. In the UI, the track path is shown in red on the segments where the speed limit was exceeded, whereas the normal speed range is plotted in green.
The complete solution consisted of features such as
- Updating the locations in real time and showing the route a device has travelled
- Showing visual indicators to represent the status and for alerts
- Displaying and plotting useful information, such as location, speed, etc.
In this solution, while some features are implemented using Siddhi Query Language, more complicated features are implemented as extensions for Siddhi, exploiting the extensible nature of WSO2 CEP. This allows users to write custom operators to implement more complex and domain-specific functionalities to be built into the CEP server.
Using machine learning models to analyze event streams
Data mining and machine learning is concerned with the study and implementation of models that can learn from past data and then identify patterns.This is a fast growing aspect in the field of data mining. Machine learning methods in particular proved to be very effective when analyzing large volumes of data such as stock market analysis and predictions, computer vision applications and bioinformatics.
Integrating WSO2 CEP with machine learning models enables more advanced monitoring mechanisms that can be used and lets the user identify and capture more complex patterns of incoming events streams in real time.
Predictive Model Markup Language (PMML) is an XML-based format of defining models developed by the Data Mining Group. In this solution CEP used PMML to built machine learning capabilities into it. The ‘R’ software is used to create a machine learning model. The model is then converted into the PMML. Finally, an extension is written to WSO2 CEP, which can decode PMML and execute the model against incoming event streams. The following is a neural network that was developed and converted into PMML, and then executed through a CEP extension for testing the functionality:
The highly extensible nature of WSO2 CEP makes it possible to write extensions easily. Once the extension is written, they can be used to write Siddhi queries just like the built-in Siddhi operators.
Queries like the following can be written to analyze a stream of events with a machine learning model:
from pmmlInputStream#transform.mlearn:getModelPrediction ("/home/Project/ NeuralNetwork.pmml", A_follower_count, … , B_network_feature_3) select response, Predicted_response insert into predictedStream
This query will send the incoming events in ‘pmmlInputStream’ through the neural network represented by ‘NeuralNetwork.pmml’ and the output produced by the model is emitted as the predictedStream.
WSO2 CEP identifies the most meaningful events within the event cloud, analyzes their impacts, and acts on them in real time. Built to be extremely high performing and massively scalable, it offers significant time savings and affordable acquisition. WSO2 CEP is 100% open source and is architectured in a highly extensile manner. This extensible nature of CEP allows organizations to inject domain-specific knowledge into the CEP server and build up more effective and comprehensive solutions.
- Sajith Ravindra
- Senior Software Engineer