Making Real-Time Applications Simpler with WSO2 Stream Processor

  • By Sriskandarajah Suhothayan
  • 11 Jan, 2018

With rapidly changing business environments, processing business events on the fly is vital for any organization to stay competitive. Currently the complexity of code and complex multi-node deployments along with the inability to change systems quickly hinders enterprises from reaching the potential of real-time stream processing. WSO2 takes real-time stream processing to the next level by making it much simpler with its next generation analytics product - WSO2 Stream Processor.

This article discusses the challenges of first-generation stream processing solutions and further presents how WSO2 Stream Processor is implemented, the problems it solves, and why it should be your choice for real-time stream processing.



The pace of business innovation is increasing rapidly in today’s highly competitive environments. To achieve a competitive edge and win customers, organizations need to react and respond quickly to new market trends as well as customer behaviors and environmental patterns. It is critical to have capabilities to process data as it comes in and take decisions swiftly within milliseconds. Stream processing (or streaming analytics) provides the ability to make real-time decisions and is an important and integral part of an agile business.

Stream processing systems allow you to understand the environment by consuming events, processing them in real-time, building summarizations, and generating real-time alerts. This helps organizations to react to various market events instantly, giving them a competitive edge to stay ahead.

Why Has it Been Hard to Build Stream Processing Applications?

First generation streaming analytics products, such as Apache Storm, are built to help developers implement stream processing pipelines within general purpose programming languages. This gives developers the flexibility to implement anything and everything they require. It also provides the platform to deploy the implemented code in a distributed manner, such that, they can run stream processing at scale.

However first generation stream processors present two challenges:

  1. Present day enterprises have continuous changing requirements to adapt to. As such, it's hard to write stream processing logic in traditional languages. Changing requirements create the need for changes and revising of code, which is a time-consuming task.
  2. In order to run your streaming application, most existing stream processing systems available in the market need up to 5 to 6 nodes to set up a production-ready deployment. This is a complex and inexpedient process to maintain.

How Does WSO2 Stream Processor Make it Simpler?

The objective was to solve stream processing challenges by building a system that can help developers write scripts much faster and in a simpler manner, which they can also change as and when there is a change request. Where possible we give control to business users to fine-tune their systems independently, such as when managing rules, thereby optimizing the day to day business operations with seamless and agile processes.

WSO2 Stream Processor collects events with multiple messaging formats via multiple transports. It uses streaming SQL to process streams, detect complex events patterns and make predictions using machine learning models. It can also generate and notify the processed results as alerts instantaneously and visualize them via real-time interactive, and user-friendly dashboards.

WSO2 Stream Processor reduces the complexity of deployment by letting you run with just 2 commodity nodes, providing high availability and allowing any organization to become competitive in the marketplace by utilizing stream processing.

WSO2 Stream Processor is built as a lightweight, open source, high performance, stream processing platform which understands streaming SQL queries in order to capture, analyze, process and act in real time. This facilitates real-time, intelligent, actionable business insights, and data products. With the WSO2 Stream Processor’s simple deployment and it’s ability to adapt to changes rapidly, enterprises can now go to market much faster and achieve greater ROI.

The WSO2 Stream Processor simplifies the Stream Processing Application lifecycle in three ways:

1. Faster Development

Developers now have the right tools to build reliable and high performing streaming applications much faster. This is achieved by the state of the art Siddhi stream processing and complex event processing engine which lies at the core of WSO2 Stream Processor. It also includes:

  • Easy to use streaming SQL language, as developers are already familiar with SQL and databases.
  • A powerful toolkit of stream processing operators via Siddhi such as, filtering, window operations, aggregations and summarizations, pattern machine, and event correlations.
  • Up to 50+ extensions to support more complex use cases like geographical processing, natural language processing, machine learning, anomaly detection and others.
  • Support for consumption of events and publishing alerts through connectors available out of the box for well-known protocols such as HTTP, Email, Kafka, MQTT, JMS, TCP, File supporting XML, JSON, Text, binary and Key-Value messages.
  • Out of the box integration with popular data storage systems such as RDBMS databases, Apache Solr, MongoDB, Apache Hbase, Hazelcast, and many more.

The following snippet explains how a user can calculate the total hourly cookie production, on each event arrival, at each production line, and get an alert if it has dropped below 5% within the last 30 minutes.

Sample Siddhi Query

Figure 1: Sample Siddhi Query

To make developers' lives even easier, there’s a Stream Processing Studio which has auto-completion, syntax highlighting and provides integrated documentation help while creating streaming applications. Simulations support is provided to test the application using random data, csv or by playing back from databases as and when they develop the applications. Debugging support is also provided to help developers to debug if they have made logical errors during development.

Stream Processing Studio

Figure 2: Stream Processing Studio

2. Ease of Deployment

One of the most important aspects of the application lifecycle is deployment. Deployment is made simple with only 2 nodes as the minimum number of nodes required for processing. 2 nodes are required as all production deployments need at least 2 nodes to achieve high availability. However, this simplification of the deployment should not come at the cost of performance. To ensure this does not happen, the Siddhi Stream Processing Engine is embedded within the WSO2 Stream Processor. This provides the ability to process more than 2M events per second within the JVM and process up to 900K events per second over the network for simple streaming queries. It can also effortlessly handle more than 100k events per second for complex event processing scenarios.

Of course, there are use cases that extend beyond 100k events per second benchmark in scenarios such as IoT analytics and device monitoring. For such use cases connecting multiple stream processing nodes together into a cluster using Apache Kafka is possible. With this, WSO2 Stream Processor provides a platform that lets you do low latency, event processing with zero downtime that can go beyond 30+ billion events per day.

Since stream processing systems are stateful, traditional stream processing systems force developers to redo applications to achieve scalability. To overcome this, the WSO2 Stream Processor is built to be intelligent enough to decide on how it should deploy in a scalable manner. For users who are familiar with the system and want to take control, they can annotate the Siddhi applications to provide more concise information on how it should be scaled without the need to rewrite the application.

A challenge encountered by DevOps is migrating configurations when it is essential to upgrade the system for new features or bug fixes. This obstacle is resolved with the WSO2 Update Manager (WUM) and the WSO2 Stream Processor's single configuration file approach. All you need to do is update the pack with WUM and replace a single configuration file and you will be ready to roll out.

Status Dashboard

Figure 3: Status Dashboard

Distributed systems are inherently prone to failure, and the WSO2 Stream Processor is not an exception. There are logics for auto recovery and redeployment as nodes join and leave the cluster. Without a proper view of the cluster and understanding the details, a DevOp will not be able to isolate the components that have issues and take proactive actions. Observability to the system is enabled by a status dashboard that monitors stream processing nodes by not only monitoring their JVM health but also by deeply analyzing the performance of each query element that is deployed.

3. Self Service and Omniscience

Business users hate raising a change request and waiting for days for developers to deliver them. To solve this pain, the WSO2 Stream Processor facilitates collaborative deployments where the developer who builds the solutions can give business users access to modify the processing logic by using pre-accepted configurations, parameters, and simple conditions with appropriate validations. In doing so the lives of developers and business managers become much simpler because now their systems give access to the business managers in order to change the execution logic themselves without the need of a lengthy change request process. The ease of development is also taken to the next level by automating the deployment of the managed rule as and when the business user has configured them.

Managed Business Rules UI

Figure 4: Managed Business Rules UI

Any analytics application cannot be fulfilled without the ability to present a comprehensive view of what's been analyzed. WSO2 Stream processor is packed with a customizable dashboard that allows a business user to compose their own views. It is also equipped with a widget generation wizard that helps them to build their own charts to understand their data better. When it comes to accessing data, all organizations have their privacy rules, and WSO2 Stream Processor enforces such requirements via the role-based access control of the dashboard.

Interactive Dashboard
Interactive Dashboard

Figure 5: Interactive Dashboard

As different organizations have their own preferences in visualizing their data and the need to access raw data to integrate them with their existing systems for further processing, WSO2 Stream Processor is built with a rich set of REST APIs to support out of the box data and system integrations.

Let’s Get Started

With WSO2 Stream Processor, you can get started and create real-time streaming applications in less than 10 minutes. Please refer to the Quick Start Guide to get started.

WSO2 Stream Processor is equipped for building complex, production ready, stream processing and analytics solutions faster as well as deploying and managing such solutions easily. For more details and to try out its various capabilities try its tutorials.

Happy Stream Processing!

Applies to

About Author

  • Sriskandarajah Suhothayan
  • Director
  • WSO2