All posts by Vishva Ahangama

WSO2 Stream Processor Named a Representative Vendor in Gartner’s 2018 Market Guide for Event Stream Processing

Owing to the rapid increase in innovation in today’s highly competitive business environments, agile digital organizations require capabilities to quickly respond to new market trends, customer behaviors, and environmental patterns. WSO2 Stream Processor, with its use of big data, streaming, and historical and predictive analysis, allows organizations to gain a competitive edge. It allows businesses to analyze large data streams, react to changes in real-time, predict trends, and even open up new, domain-specific avenues for entirely new services and products.

As validation of WSO2’s strategy to deliver an open source, cloud native, and lightweight stream processing product that can easily scale up to handle large event rates, WSO2 Stream Processor was recently listed as a Representative Vendor in Gartner’s Market Guide for Event Stream Processing1. The guide looks at vendors with cloud and on-premises offerings, as well as proprietary and open-source products.

Gartner defines event stream processing platforms as, “software systems that perform real-time or near-real-time calculations on event data ‘in motion.’ The input is one or more event streams containing data about customer orders, insurance claims, bank deposits/withdrawals, tweets, Facebook postings, emails, financial or other markets, or sensor data from physical assets such as vehicles, mobile devices or machines.” The report identifies two sub-segments in the market: stream analytics event stream processing (ESP) offerings and stream data integration ESP offerings.

According to the report, “Increasingly demanding consumers and intensifying digital competition are pushing analytics from transactional to continuous. To achieve the necessary continuous intelligence, data and analytics leaders must understand and master the event stream processing market.”

WSO2 Stream Processor helps agile digital organizations accelerate time to insight from data by providing users more autonomy in managing tools. It is an open source, cloud native, and lightweight stream processing platform that understands streaming SQL queries in order to capture, analyze, process and act on events in real time. This facilitates real-time streaming analytics and streaming data integration. With the product’s simple deployment and its ability to rapidly adapt to changes, enterprises can go to market faster and achieve greater ROI. It comprises many features that enable enterprises to build streaming analytics capabilities and derive meaningful insights out of an organization’s data.

Unlike other offerings, it is the only stream processor that provides high availability and 100K+ throughput with just two nodes and scales to 30+ billion events per day with its Kafka-based distributed deployment. The Siddhi Streaming SQL language also enables users to adapt to the market faster with quicker development times. The product can be utilized as a lightweight deployment, with the ability to handle large event rates, sophisticated and complex streaming operators, or out-of-order message processing. It can also process real-time queries that span from seconds to years. With its rich and agile development experience, it also provides business users more autonomy in managing tools.

1Gartner, Inc. “Market Guide for Event Stream Processing” by Nick Heudecker and W. Roy Schulte, June 15, 2018.

Ask an Expert: Catching up with Srinath Perera

Srinath Perera is vice president of research at WSO2. He is a scientist, software architect, author, and speaker. He is also a key architect behind Apache Axis2 and WSO2 Stream Processor. We caught up with Srinath recently to get his take on the significance of Streaming SQL, the future of open source stream processing solutions, and why we must learn to think, question, and see beyond the obvious.

1. What has your journey at WSO2 been like?

This is my ninth year at WSO2, but I have been working with Sanjiva Weerawarana on similar technologies since 2003. Yes, it’s been close to 15 years, and it’s been a lot of fun. I have worked on a wide variety of challenging problems, and have worked with many brilliant individuals who will make good stories for one’s grandchildren one day. I have done a lot more than I imagined years ago.

2. For agile digital businesses, the availability of business insights is a significant factor in gaining a competitive advantage. How does WSO2 Stream Processor help?

Our product can easily plug-in to a user’s system and collect data. You could then write queries using Streaming SQL to detect important conditions. Streaming SQL is similar to SQL, but works on data streams instead of data tables. The former is flowing, while the latter is stored on a disk.

Compared to what our competitors offer, we have very powerful Streaming SQL with operators most others do not have. We enable you to use machine learning models within Streaming SQL itself. Also, if you are looking for a small deployment, our server can run a HA deployment with only two nodes and process about 100,000 events/second. If you are looking for a large deployment, we can run on top of Kafka. In the event you are unsure or undecided, you can always start small and later switch to Kafka without changing any code.

Streaming SQL is similar to SQL, but works on data streams instead of data tables. The former is flowing, while the latter is stored on a disk.”

3. What does the future hold for open source stream processing solutions?

In my opinion, stream processing has not become mainstream yet. People are still figuring out analytics. It’s not easy to find developers who excel in analytics. Stream processing has to wait for that adoption to play out. No one will try to do real-time before they figure out basic analytics; that is unless you have specialized use cases such as for stock markets, surveillance, and anomaly detection.

People are still figuring out analytics. It’s not easy to find developers who excel in analytics. Stream processing has to wait for that adoption to play out.”

4. What are the benefits of an open source stream processing solution?

I think there’s a growing trend for middleware as an open source model. They use complex code, support a wide variety of use cases, and are used by many. We are increasingly made aware that products are best built using the open source model. I think there’s no better testament than Microsoft, a company that hated open-source, but has now embraced it.

I think there’s a growing trend for middleware as an open source model. They use complex code, support a wide variety of use cases, and are used by many.”

5. How did you start working in stream processing?

A long time ago, in 2007, while I was doing a Ph.D, we worked on a paper comparing Complex Event Processors (or CEPs, which is an older name for stream processing) and rule-based systems. I was fascinated by the technology, and after I joined WSO2, I supervised an undergraduate thesis project to build an open-source CEP engine. This was in 2011 – well before stream processing became cool! It was called WSO2 Complex Event Processor back then and was later renamed WSO2 Stream Processor.

6. What is your proudest accomplishment in recent times?

In general, it is the role I have played with Apache Axis2. However, if you want me to choose something recent, I suppose my work with the WSO2 Research Team stands out. Some good work will be made public soon. I have also worked with Paul Fremantle, WSO2’s CTO, to build a framework to evaluate different emerging technologies. You will hear more about this too soon.

7. What advice would you like to give a budding developer or an architect to better their career?

I would say learn to think, question, and see beyond the obvious.”

There is this quote that I love, “Wisdom is tolerance of cognitive dissonance.” It took me awhile to understand what it meant. We all interpret how the world works, but when we discover things that do not match our way of thinking, we ignore them. However, the world is more complicated than that. By understanding those mismatches and by learning through struggle and discomfort, we achieve true wisdom. That is what that quote conveys.

I would say learn to think, question, and see beyond the obvious. I refuse to tell people I work with how to solve something. Instead, I tell them, “Tell me how you will solve it and then I will complain.” I think they are used to it now. That way, we all use put our critical thinking skills to good use and one day, they will not need me for guidance.

To learn more about Srinath’s work, follow him on Twitter and read his blog.