2018/01/11
 
11 Jan, 2018 | 3 min read

WSO2 Stream Processor: Making Real-time Stream Processing Available to the Masses

  • Vidura Gamini Abhaya
  • Senior Director - Solutions Architecture - WSO2

Today we are thrilled to announce the availability of the WSO2 Stream Processor, our lightweight, open source, high performance, stream processing platform which helps create real-time, intelligent, actionable insights for your digital business.

A significant competitive advantage for any modern businesses is the availability of business insights and information to make real-time decisions. The speed at which we collect, analyse, draw insights from an organization’s data and the time taken to respond to them, determines who ends up being the winners and losers.

The Rise of Real-time Stream Processing

Digital Transformation has seen many businesses opening up systems to others through APIs, supporting multiple ways of authenticating users and integrating multitude of systems together into a single digital platform. As the number of systems and usage goes up, it becomes impossible to keep the systems running and ensuring availability without real-time monitoring. Consumer centric digital businesses too customize experiences based on insights on buying patterns. With increased usage, fraudulent patterns, and security threats need to be monitored and acted upon.

Most systems generate such streams of events that can be transformed into valuable business insights. These events need to be collected, filtered, grouped and pattern matched in the process of transformation. Real-time stream processing technologies enable this transformation of simple event data into useful business insights. It plays the important role of a catalyst for digital transformation of modern businesses.

Adoption Challenges

There are however many challenges that enterprises face when adopting capabilities to quickly capture, analyze and process data, and act in real time.

With first generation stream processing products you had to write code and implement complex operators such as time windows, aggregations and patterns with minimum tooling support. Developing such code as well as adapting it to changing requirements is both complex and expensive. Moreover, they are inherently complex in their deployments, consisting of 5 - 6 nodes even for the simplest use cases. Such large deployments are difficult to manage and they incur high maintenance costs.

The use of streaming analytics is therefore a challenge for most businesses without the highly technical skillset and the cost involved.

The next generation of streaming analytics products solved some of these problems. Most of them support a more business user friendly SQL like language. Deployments though, still continue to be 5 - 6 nodes depending on the levels of throughput required. This makes it challenging for mainstream enterprises to adopt real-time streaming processing.

Taking Real-time Stream Processing to the Masses

WSO2 Stream Processor (WSO2 SP) is packed with features that enable any enterprise to build streaming analytics capabilities and derive meaningful insights out of the organization's data. It is powered by Siddhi, the leading open source stream processing project that has been used by the likes of Uber, Transport for London (TFL), and Experian. The streaming SQL capabilities and in-built editor have event simulation and debugging support that can help you create real-time streaming applications faster than first generation products.

The high performance and low footprint also leads to more agile deployment: it is the only competing product that can handle 100K events per second in a high-availability deployment with just two commodity servers. This 2 node setup with minimum high availability achieves enough throughput for most of your stream processing needs. We’re talking over 8 billion messages per day!

In addition WSO2 Stream Processor includes new features that makes complex aggregations much simpler to write. The new rule management console, together with React-based dashboards, make rule management and real-time visualisation accessible to any organisation that wants to harness real-time analytics to gain competitive advantage.

WSO2 Stream Processor Reference Architecture

Here’s a snapshot of some other key features of WSO2 Stream Processor:

  • Supports massive scale when deployed in conjunction with Apache Kafka
    • Demonstrated in production at 30 billion messages per day
  • Updated Siddhi Streaming SQL 4.0 language adds incremental processing support for more efficient analytics
  • Simplified time based aggregations - write a single Siddhi statement that aggregates at multiple time intervals
  • Predictive Analytics through traditional and streaming Machine Learning
  • In-built IDE, event simulator and templates for developers
  • Monitor your deployment through a status dashboard
  • Deploy business rules through a graphical UI
  • Multiple data center support
  • Leverage Edge Analytics through small footprint deployment options
Visit our product page to try out the new release yourself, and let us know if you have any feedback.
Undefined