Streaming analytics allows users to monitor event streams and respond to complex conditions within milliseconds. For example, a bank may use streaming analytics to monitor transactions and trigger an alert when a suspicious transaction takes place.
What this means is that streaming analytics essentially enables users to respond to changing conditions within seconds to milliseconds. Hence, it's also classified under real-time analytics.
Streaming analytics comes in many forms ranging from detecting simple conditions (e.g. temperature is greater than 30oC) to complex pattern detection (e.g. triple bottom pattern has been seen over Apple trading data). Refer to the blog post Patterns for Streaming Real-time Analytics for more details.
For a reader who would like to understand the different flavors of streaming analytics, the Gartner report titled The Five Levels of Stream Analytics1 released in February 2017 will be useful.
The Gartner report1 first makes several observations in its 'Recommendation'; excerpts from the report are as follows:
"While event stream processing isn't new, there is a resurgence of interest in speeding up time to insight for business intelligence and analytics processes"
"Despite increases in both the demand for stream analytics and the supply of available technology, today's stream processing deployments remain on the fringe of most IT departments"
The Gartner report1 outlines five maturity levels for streaming analytics, which provide a roadmap for organizations to follow. The report introduces five maturity levels that have been summarized in Figure 1 below.
Figure 1: Maturity Roadmap for Stream Analytics
This white paper discusses how enterprises can implement each of these levels using the WSO2 Analytics Platform to derive value from streaming data.
2. Implementing Level 1: Ingesting
According to the Gartner report1, in the 'Analysis' section, at this level, "efforts are aimed at ingesting the data as it arrives; sometimes applying data integration techniques in real time (ltering, enrichment, complex-event processing [CEP] for feature generation/abstraction); storing the data in a data lake or other data store; or conducting the analytics on the data at rest at a later time."
WSO2 Stream Processor (WSO2 SP) is an open source stream processing platform. It can ingest data from Kafka, HTTP requests, message brokers and you can query data stream using a “Streaming SQL” language. With just two commodity servers it can provide high availability and can handle 100K+ TPS throughput. It can scale up to millions of TPS on top of Kafka.
For example, the following query filters the stream to only keep events with the condition price > 100 and reshapes the stream.
from inputStockStream[price > 100]
select symbol, price * volume as transactionAmount insert into outputStockStream
Please refer to this link for more information
The following are some preprocessing actions that are possible with WSO2 SP:
Reshaping and transforming events (e.g. bring events in multiple schemas to one schema)
Enriching events (e.g. add customer ID to record after looking up a data base)
Aggregation (e.g. when the data rate is high, only keep per second or per hour summaries)
Fix missing data (e.g. imputing replacement value), detect and flag errors, error correction
Join event streams and correlation
Ingesting can cleanup the data and enforce a shared schema. We believe this can make the rest of the processing simple and avoid each data processing task to repeat the cleanup logic. Furthermore, it stops different tasks from using different definitions of cleanup data.
Data ingesting that's possible with WSO2 SP are not limited to the above use cases. A good example of a smart ingesting system can be found in a railway sensing system, where each part of the railway is monitored with overlapping sensors. This allows the system to detect malfunctioning sensors and automatically fixes the readings as part of the ingesting system.
3. Implementing Level 2: Orienting
According to the Gartner report1, in the 'Analysis' section,, "at this level, the enterprise, or more likely specifIc business units, recognizes the uses for stream processing for situation awareness around marketing, customer support or operations monitoring." In our view, this may include a dashboard with visualization to show the current status in context to the past.
WSO2 SP can be used to build real-time situational dashboards. For example, consider the following use case where video uses data streams collected from buses in London (via Transport for London) to track, monitor, and visualize their operations. Refer to WSO2 TFL demo for more details.
In another example, video uses data streams collected from a real football game through sensors deployed in the ball, player's boots, and goalkeeper's gloves to track the game in real time and to provide high level event statistics, such as ball possession time and detecting offsides. Refer to our Football Analytics use case story for more details.
The following are a few more sample use cases that are possible with WSO2 SP:
Operation dashboards showing progress towards a goal
Drug interaction warnings for a doctor in prescription view
Risk analysis views for a portfolio and alerts
Logistic monitoring and optimization views
Location analytics and tracking
As mentioned earlier, a key aspect of this level is to present the data in context to make it easier to have situational awareness. Among examples of this are alerts, color coding to show what's normal and abnormal, past averages with data, etc.
Furthermore, for some domains, such as finance and banking and smart energy, WSO2 provides out-of-the-box solutions. Refer to https://wso2.com/analytics-and-stream-processing/solutions/ for more details on these solutions.
4. Implementing Level 3: Advising
According to the Gartner report1, in the 'Analysis' section, "at this level, the stream processing is extended with model-based decision support capabilities."
To illustrate the implementation of this level, let us consider the WSO2 Fraud and Anomaly Detection Solution. This solution provides a system that may detect a fraud and raise an alert, allowing the operator to go to a view that shows the incident and data around the incident to make the final decision and carry out the task from the same view itself.
Moreover, most systems would provide prescriptions that are potential solutions, further aiding human decision.
The following are more examples where WSO2 SP can be used to build an advising solution:
5. Implementing Level 4: Deciding
According to the Gartner report1, in the 'Analysis' section, "the focus at this level is improving capabilities to ensure reliability, quality and fault tolerance or to accelerate the time to respond to emerging situations because a human is not required in the loop."
We believe decisions systems are used in various situations as follows:
When outcome has very high confidence
When downside of a failure is not apparent or negligible (e.g. production recommendation, advertisement)
Things happen too fast to include a human in the loop
At the same time, we feel these algorithms need to be carefully monitored as they can have unexpected consequences. Among some examples are 23 million book in Amazon and stock market flash crashes. Hence, these solutions must be carefully designed and tested.
The following are some examples where WSO2 SP has been or can be used to provide deciding solutions.
Advertisement buy and placement
Targeted offers for smart grid and other energy optimizations, such as Next
Furthermore, for some domains such as finance and banking, and smart energy WSO2 provides out-of-the-box solutions.
6. Implementing Level 5: Learning
According to the Gartner report1, in the 'Analysis' section, "at this level, the organization's streaming data and analytics capabilities use machine learning to adapt to changing market and environmental conditions."
With WSO2 SP a user can run machine learning models which are exported as Predictive Modeling Markup Language (PMML). Most R and Python models can be exported as PMML which can then be loaded into WSO2 SP to run predictive analytics.
A sample query to evaluate the model using CEP looks as follows:
select T, V, AP, RH, prediction
insert into data_output
You can find detailed instructions from
The following are two learning systems that are built with WSO2 SP.
The following are a few specific example learning systems that you can built with WSO2 SP:
Ranking (e.g. customers)
Sequence detection, pattern detection
Sentiment analytics and text analytics
Streaming analytics comes in many flavors and its adoption significantly differs from case to case. Gartner's five maturity levels enable users to evaluate the maturity of their streaming use cases and provides a path to make full use of streaming analytics. By incorporating these, we believe enterprises can address the business challenges faced with the adoption of technology that's required to remain competitive and act fast on new business opportunities. They particularly help organizations to find that balance between their technology and business strategies.
This paper explains how each of these levels can be implemented with WSO2 SP, a powerful open source analytics product that analyzes data streams in real time.