Tag Archives: Analytics

WSO2 Stream Processor: Making Real-time Stream Processing Available to the Masses

Today we are thrilled to announce the availability of the WSO2 Stream Processor, our lightweight, open source, high performance, stream processing platform which helps create real-time, intelligent, actionable insights for your digital business.

A significant competitive advantage for any modern businesses is the availability of business insights and information to make real-time decisions. The speed at which we collect, analyse, draw insights from an organization’s data and the time taken to respond to them, determines who ends up being the winners and losers.

The Rise of Real-time Stream Processing

Digital Transformation has seen many businesses opening up systems to others through APIs, supporting multiple ways of authenticating users and integrating multitude of systems together into a single digital platform. As the number of systems and usage goes up, it becomes impossible to keep the systems running and ensuring availability without real-time monitoring. Consumer centric digital businesses too customize experiences based on insights on buying patterns. With increased usage, fraudulent patterns, and security threats need to be monitored and acted upon.

Most systems generate such streams of events that can be transformed into valuable business insights. These events need to be collected, filtered, grouped and pattern matched in the process of transformation. Real-time stream processing technologies enable this transformation of simple event data into useful business insights. It plays the important role of a catalyst for digital transformation of modern businesses.

Adoption Challenges

There are however many challenges that enterprises face when adopting capabilities to quickly capture, analyze and process data, and act in real time.

With first generation stream processing products you had to write code and implement complex operators such as time windows, aggregations and patterns with minimum tooling support. Developing such code as well as adapting it to changing requirements is both complex and expensive. Moreover, they are inherently complex in their deployments, consisting of 5 – 6 nodes even for the simplest use cases. Such large deployments are difficult to manage and they incur high maintenance costs.

The use of streaming analytics is therefore a challenge for most businesses without the highly technical skillset and the cost involved.

The next generation of streaming analytics products solved some of these problems. Most of them support a more business user friendly SQL like language. Deployments though, still continue to be 5 – 6 nodes depending on the levels of throughput required. This makes it challenging for mainstream enterprises to adopt real-time streaming processing.

Taking Real-time Stream Processing to the Masses

WSO2 Stream Processor (WSO2 SP) is packed with features that enable any enterprise to build streaming analytics capabilities and derive meaningful insights out of the organization’s data. It is powered by Siddhi, the leading open source stream processing project that has been used by the likes of Uber, Transport for London (TFL), and Experian. The streaming SQL capabilities and in-built editor have event simulation and debugging support that can help you create real-time streaming applications faster than first generation products.

The high performance and low footprint also leads to more agile deployment: it is the only competing product that can handle 100K events per second in a high-availability deployment with just two commodity servers. This 2 node setup with minimum high availability achieves enough throughput for most of your stream processing needs. We’re talking over 8 billion messages per day!

In addition WSO2 Stream Processor includes new features that makes complex aggregations much simpler to write. The new rule management console, together with React-based dashboards, make rule management and real-time visualisation accessible to any organisation that wants to harness real-time analytics to gain competitive advantage.

WSO2 Stream Processor Reference Architecture

Here’s a snapshot of some other key features of WSO2 Stream Processor:

  • Supports massive scale when deployed in conjunction with Apache Kafka
    • Demonstrated in production at 30 billion messages per day
  • Updated Siddhi Streaming SQL 4.0 language adds incremental processing support for more efficient analytics
  • Simplified time based aggregations – write a single Siddhi statement that aggregates at multiple time intervals
  • Predictive Analytics through traditional and streaming Machine Learning
  • In-built IDE, event simulator and templates for developers
  • Monitor your deployment through a status dashboard
  • Deploy business rules through a graphical UI
  • Multiple data center support
  • Leverage Edge Analytics through small footprint deployment options

Visit our product page to try out the new release yourself, and let us know if you have any feedback.

Why Swiss Chocolate Relies on WSO2

The Swiss Federal Office of Information Technology, Systems and Telecommunication (FOITT) is one of the internal ICT service providers in the Federal Administration. It supports the administration by developing and providing efficient, secure, and user and public-friendly IT solutions. As part of its responsibilities, FOITT manages more than 40,000 enterprise users of two of their key platforms – one an electronic customs declaration process for imports/exports and the other, an automated way to manage revenue from taxes.

While these platforms have proved successful, FOITT embarked on a digital transformation initiative to make these more efficient. What they had hoped to achieve was the ability to scale to provide a more seamless experience to users.

At WSO2Con US 2017 Dr. Gion Sialm, chief architect at FOITT, explored how they leveraged WSO2 technology to achieve their objectives. They worked together with Yenlo, a WSO2 Premier Certified Partner, to implement their solution. To illustrate how the two platforms work, Gion took the example of Swiss chocolate – the process of importing cocoa to make chocolate and the distribution of the end product within and outside Switzerland.

The e-Dec (Electronic Declaration) Platform

All goods, and in this instance the import of cocoa and export of chocolate, need to be declared and there’s a specific process that needs to be followed. Given that it’s a fairly complex process involving many functions and stakeholders, FOITT created the e-Dec platform to simplify this process. What it essentially did was digitize this process and made it more efficient and user-friendly. As with any digital platform, the e-Dec platform too needed to be refreshed and revamped to be more aligned with new requirements.

For instance, the platform had a lot of different protocols and some were extremely outdated like POP3S and FTPS. Apart from this challenge, the application was based on the Oracle WebLogic Server, which follows the eXtended Architecture (XA) pattern. “Previously, WSO2 products didn’t support XA, but because of FOITT’s requirement, it’s now a part of their feature list,” noted Gion.

The Fiscal-IT Platform

On the retail side, all goods, like chocolate, sold within Switzerland carries a value-added tax (VAT). Previously, these transactions were done manually so FOITT built the Fiscal-IT platform that automated this process. Again, like the e-Dec platform, this too required improvements to further streamline this process.

For instance, the platform was created in a modular manner so as to have the best of breed technology for each feature resulting in a mix of multiple different technologies, like FileNet, Java and SAP, which all needed to be integrated. “Because we decided to employ microservices, we ended up with a lot of REST and SOAP APIs as well as JMS so we needed an enterprise service bus that was flexible enough to maintain these things easily,” said Gion.

The WSO2 Solution

They followed the same architecture for both platforms so as to reduce cost and speed up their go-to-market. The API Gateway, Publisher and Store components of the WSO2 API Manager as well as the WSO2 Identity Server as the Key Manager were used as their core API management solution. WSO2 integration technology was used for routing and message transformation between the sender’s and receiver’s different protocols. WSO2 analytics (not pictured in the architecture diagram above) also plays an important role in the solution — FOITT, together with their service providers, developed a dashboard using WSO2 Data Analytics Server to identify any problems that occur in the application. The user just has to type in the source and destination program and within a few seconds the metadata of all the messages is collected (message tracing) so that errors can be easily identified. The dashboard can even correlate the messages with the log files, which is a very important feature in a distributed landscape like this.

“WSO2 products relate to digital transformation like the Swiss army knife relates to MacGyver. Our platforms are evolving rapidly. In order to keep pace with this innovation it’s important to have a strong relationship and collaborate well with WSO2,” says Gion. “Automation is also key. We have to manage 11 stages throughout our platforms and doing it manually would be quite impossible,” he adds.

To learn more about how FOITT is leveraging WSO2 technology for key government initiatives, watch Gion’s presentation at WSO2Con US 2017:

Bringing an Efficient Home Care Solution to Life with WSO2 Technology

Senior citizens and disabled people—many in fragile health and requiring assistance—often have limited resources for managing their health and ensuring their security. Effective home care solutions allow such people to safely go about their day-to-day lives and enhances their quality of life. To aide home caregivers and patients, Raffaello Leschiera, a solution architect at Engineering Ingegneria Informatica, proposed a reference architecture for efficient home care using WSO2 technology at WSO2Con EU 2017.

Raffaello began by exploring the proposed reference architecture that connected and interfaced with all stakeholders, like the patient, his/her family and medical staff. Firstly, they need to collect data from medical devices in the patient’s home. Protocols like IEEE VU specifications are used and medical devices are mediated using Arduino and Raspberry Pi boards. Once collected, the data needs to be normalized and stored so it’s represented in the same way no matter which device it was collected from.

This data needs to run through analytics to monitor the patient’s health, process events and if needed, send notifications through various communication channels. Data integration channels using the HL7 standard protocol for health care is used to send this data to medical staff. The medical staff can then access it through web and mobile interfaces and an API gateway decouples all features from these user interfaces. And finally, the entire system needs to be synchronized and controlled by identity and access management to ensure security and privacy.

Reference architecture for a home care solution

Raffaello noted that WSO2’s comprehensive technology platform, particularly its integration and analytics capabilities, were the main reasons for picking WSO2 as their technology partner. The open source nature of the products was also a key deciding factor since Raffaello and his team work with many public administrators who prefer to adopt solutions that are completely open source. “WSO2 has a wide technology platform so you can find the right answer to every part of your problem,” said Rafaello. “And because all the products seamlessly integrate with each other it’s easy to focus on the domain problem rather than the technology problem,” he added.

To describe how WSO2 products were used for different tasks, Raffaello compared the home care solution to a football game:

  • Goalkeeper: WSO2 Microservices Framework for Java (WSO2 MSF4J) serves as the goalkeeper. This is the entire back-end of the system, which is based on lightweight microservices that are developed, deployed and monitored through MSF4J in a highly scalable and reliable manner with integrated security.
  • Defenders: WSO2 Data Analytics Server serves as one defender that receives data, analyzes it in real-time, and sends notifications. WSO2 Enterprise Integrator is the next defender who transforms disparate types of data into a normalized format and sends it to the hospital IT systems.
  • Forwards: WSO2 API Manager is one of the forwards, which faces the medical staff and is used to design, prototype and publish APIs and govern API usage. WSO2 IoT Server is another forward, which faces the medical devices for data collection, device management and protocol support.
  • Wings of the pitch: Here the WSO2 Identity Server takes care of all the strict security and privacy requirements.
  • Center of the pitch: Finally, WSO2 Governance Registry serves as the ‘Lionel Messi’ at the center of the pitch; in other words it governs the solution through surveillance just like how Messi would guide and lead his team to victory.
  • For this solution to work, Engineering Ingegneria Informatica needed a remote device that can track a patient’s movements within his/her home. Enter Joe Care (or the Joker pictured above). Joe Care is a remote presence device that is flexible and agile enough to move around the patient’s home. They used various technologies like Arduino boards, software that deals with movement and the sense of space as well handling (touch). It served as the medical eyes, ears, voice and fingers within the patient’s home.

    In the future Rafaello and his team aim to engage with users more, further analyze threat paths and include more technology like wearables that monitor movement and exercise. They would also like to create more intelligent early warning score models and move their entire solution to the cloud so more patients and operators can access it.

    Watch Rafaello’s presentation at WSO2Con EU 2017 below to learn more about their home care solution powered by WSO2.

A Smarter Transport Management System for London with the Help of WSO2

Transport for London (TfL) has a daily challenge – to keep a city of over 8 million people moving around the metropolis. Its magnitude can neither guarantee the transport system will always absorb commuters nor give them a congestion-free experience. It is a place where the smallest of changes would have a massive impact on your journey. Citing an example, Roland Major, a former enterprise architect at TfL, says that a London Underground strike once saw a 3% increase in traffic and a staggering 90 minute increase in journey time. Estimates project a 60% increase in congestion around central London by 2031.

Given all these complications, TfL decided to become more intelligent with technology to reduce commuter times, make the roads safer for pedestrians, cyclists and drivers, and to slow the pace of traffic. Intelligence and data with a purpose are the buzzwords here. “We need better understanding of real-time demand. What insight can we get from our data, and how can we get innovative with all this information?” says Roland. He was actively involved with TfL’s Surface Intelligent Transport System (or SITS), a project that aims to better manage the city’s entire road space of pavements, cycle lanes, and motorways.

SITS’ business proposition is that it can offer billion pounds’ worth benefit to London by identifying delays in the road networks sooner than it is done at present: “We weren’t detecting incidents, and by the time we have detected them, they were already over. With technology, we can see these incidents early. We recognized that the market can do sensible things with our data,” says Roland. For example, within the traffic light system in London, TfL manages an estimated 7,000 junctions around the city and 14,000 magnetometers detect millions of daily events. This data is discarded after analysis; however, if used, TfL realized that the response time to delays improved by 15 minutes.

TfL has a 10 year plan in place, with all the of different required components mapped out. Data analytics form the core of this operational model. Data is obtained from GPS systems and bus routes. The road incidents are logged and used to determine what additional information is needed to understand and manage each leg of commuter journeys. All the data is hosted on the cloud and currently TfL is in the process of adding these components to the framework.

TfL’s transport management system

London’s new road management system relies on WSO2’s API management, integration, identity and access management, and analytics products for the intelligent work needed. These products are deployed on a private cloud managed by WSO2. The starting point – LondonWorks, a registry of all road works and street related events, both planned and current, in the Greater London area. LondonWorks is used to assess road networks, coordinate the various road works to minimize congestion and for inspection, compliance, and monitoring. Maps and forms of type data have been integrated to allow entry of incidents into the system and their identification on the map.

As their model progresses, TfL has ambitious plans for all the data they have streaming in – big data analytics to give them more insights to road movements, which will enable them to give the necessary alerts and empower them with smarter ways to deliver better, safer commuter experiences for London.

Watch Roland’s presentation for more details on TfL’s plans for London.

Explore the WSO2 middleware platform with its offerings in API management, integration, identity and access management, analytics, and IoT.

Did you know that WSO2 won TfL’s data analytics Hackathon contest? Learn all about it.

UNRWA and Capgemini: Creating a Refugee Centric Data Model for Better Insights

The United Nations Relief and Works Agency for Palestine Refugees (UNRWA) has over 5 million registered refugees requiring education, healthcare and social safety assistance, among others. UNRWA aids refugees across five countries – namely Lebanon, Jordan, the West Bank, Syria, and the Gaza Strip which has over 500,000 students, 692 schools as of now, and hundreds of primary health facilities.

In order to automate several processes across the region, the team based in Gaza had already developed the Education Management Information System (EMIS) consisting of three modules (students, staff and premises) and reporting tools. EMIS captures information and manages the educational progress of half a million students, by integrating data from registration, health, facility management and human resources systems that are already in existence.

Yet, given the numbers and scale of its operations, a central data model that has the capacity to integrate data from several entities was the need of the hour to support its regional operations and EMIS. To transform their information management system, UNRWA and Capgemini used WSO2 technology to create a model which mirrors UNRWA’s organizational ethos – placing the refugees at the heart of all their operations.

“The technology is there, but it’s really about the people,” says Francesco Lacoboni, Managing Consultant at Capgemini. Accordingly, the main drivers of the new UNRWA Enterprise Architecture are built upon the strategic principles of people, information, collaboration, and security. People influence how the information is created, managed, and consumed. The platform is an information-centric one – rather than managing documents, it manages open data and content. Its shared approach design aims to improve collaboration, reduce costs, maintain standards, and ensure consistency across the board. Security and privacy features for data protection round off the principles of this platform.

Before the new model was introduced, there was a time where the information that streamed through the system was physically replicated via the transaction log. For reasons of ease and efficiency, UNRWA and Capgemini decided to provide a common set of APIs to all the developers, not only to fulfill the needs of the specific application, but to also create the framework for future use of this semantic concept. Every entity has a credible API that can be used to navigate the knowledge, eliminating the need to design a new API. The resultant Common Data Model (CDM) was created using OWL (Web Ontology Language), and its architecture and governance completed using WSO2’s integration and API management platforms.

For Luca Baldini, Chief of Information Management Services at UNRWA, it was the first time such an approach was used and now that it has been rolled out, he praises its benefits: “The new model has been very productive, as it created a common language between IT specialists and our business representatives. We can use different kinds of technology for data retrieval and distribution.” Francesco believes one of the main benefits of the new model is that it helps increase the transparency of UNRWA’s operations. Now that the new model is successfully in practice, analytics is the next frontier and they hope to leverage WSO2’s analytics capabilities to meet their requirements. Spurred by the possibilities of analytics, plans are in the pipeline to use this data model along with unstructured data provided from the field to improve operations and add further value.

You can watch Luca’s and Francesco’s presentation at WSO2Con USA 2017 to hear more about their project.

Learn more about WSO2’s integration, API management and analytics capabilities if you would like to use them in your enterprise.

Perfecting the Coffee Shop Experience With Real-time Data Analysis

Picture a coffee shop.

The person who runs this shop (let’s call her Sam) operates an online coffee ordering service. Sam intends to differentiate her value offering by providing a more personalized customer experience.

Offering customers their favorite coffee as they walk into the store, rewarding loyal customers with a free drink on special occasions – these are some of the things on her mind.

Further the value creation is not limited to her customers but extends to business operations such as real-time monitoring and management of inventory. Sam wants:

  • A reward system where points will be calculated based on order value. Once a reward tier point value is reached, the customer will be notified in real-time about an entitlement for a free drink
  • Inventory levels are updated in real-time on order placement. An automated notification is sent to suppliers in real-time as predicted re-ordering levels are reached

Overview of the solution


Understanding the customer is the first action in  providing a personalized experience. To do this, one must collect intelligence. In today’s digital business, customers pass through many ‘touchpoints’, leaving a digital trail. For example, many would search ‘health benefits of coffee’, some would publish a review on their favourite coffee type – and so on.           

Application Program Interfaces (or APIs) come into play here. In a business context, APIs are a way that businesses could expose their services externally, so that consumers, using an app or some technological interface, can subscribe to and access these services.

For example, Sam can have an “Order API”  that provides a way for consumers to order coffee from her shop using their mobile app.

What we now need is a simple way to create and publish said API and a central place for consumers to find and subscribe for this API. We also need proper security and an access control mechanism.

Data leaving through the API needs to be collected, stored and analyzed to identify patterns. For example, Sam would like to know what the most used combination of ‘coffee and flavors’ is, at which point of the day, by which type of users – which would be helpful for targeted promotion campaigns. For this, we need to understand the data that comes through. 

In base terms, the system requirements for developing such a solution are to: 

  • Design an API for end user access
  • Publish end user attributes (API data) for analytics
  • Process API data in real-time
  • Communicate outcomes

The solution requires to integrating API Management with real time event processing ,where the API end user attributes can be published to a steaming analytic engine for real time processing. There are many offering in the market that provides separate offering, however integrating these offering has it’s own challenges.

WSO2 offers a completely integrated 100% open source  enterprise platform that enables this kind of use case – on-premise, in the cloud, and on mobile devices.

We offer both an API management and streaming analytics product, architected around the same underlying platform, which enables seamless integration between these offerings.

WSO2 API Manager is a fully open source solution for managing all aspects of APIs including creating, publishing, and exposing APIs to users in a secure and scalable manner. It is a production-ready API management solution that has the capability of managing all stages of the API lifecycle in a massively scalable production environment.

WSO2 CEP is one of the fastest open source solutions available today, find events patterns in real-time milliseconds. It utilizes a high-performance streaming processing engine which facilitates real time event detection, correlation and notification of alerts, combined with rich visualization tools to help build monitoring dashboards.

WSO2 MSF4J is a lightweight framework that offers a fast and easy programming model and an end-to-end microservices architecture to ensure agile delivery and flexible deployment of complex, service-oriented applications.

Building an API for end user access

Let’s examine how we can build this with what we’ve listed above.

WSO2 API Manager includes architectural components, the API Gateway, API Publisher and API Store (Developer Portal), Key Manager, Traffic Manager and API Analytics. The API Publisher provides the primary capability to create and publish an API. The developer portal provides a way for subscribers to access the API.

API data to the streaming analytics engine is published through a default message flow. The solution we have in mind requires changing this default flow to capture and publish custom user data.

This is implemented  as a custom ‘data publishing mediator’ (see mediation extensions).  

In a nutshell, message mediation for simplification can be described as the inflow processing of messages, which could be modified,transformed, routed and many other ‘logics’.  Mediators are the implemented component of the logic, which when linked together creates a sequence or flow of the messages.  With API Manager tooling support, a custom flow is designed using a class mediator to decode, capture and publish end user attributes.

The custom sequence extracts the decoded end user attributes passed via JWT headers. The class mediator acts as a data agent that publishes API data to WSO2 CEP. The parameters passed to the class mediator include the connection details to CEP and the published event stream.

Real-time processing of API Data

To capture API data for real-time processing, the same stream definition and event receiver  is created and mapped to the stream. WSO2 provides a comprehensive set of extensionspredictive analytics capabilities  are added via the WSO2 ML extension.

Coffee reordering

The mechanics of reordering coffee based on a real-time analysis goes thus:

An event table represents the inventory details (‘drink name’ ‘ordered quantity’ , available quantity). The API data stream is joined with the event table and the available quantity in stock is reduced using the order quantity as and when events are received. When the reorder quantity level is reached, a email notification is published.

Real-time rewards

Similar to the approach above, the API data is joined with an event table, the event table represents the end user and the reward points generated per order. The reward points are equated to the order size and reward points are added with each new order placed. A reward limit threshold is defined, and when the limit is reached for a new order a notification is sent to the end user, offering a free drink.

Communicating outcomes

To communicate the  outcome of the real time processing event processing, WSO2 CEP provides capability to generate alerts via an SMS, email, user interface  etc. through event publishers. Email notification can be generated to alert management when re-order level are reached, as well as send an SMS to the client to notify offer for a free drink.

Meanwhile, the backend service for order processing is developed as a Java Microservice using WSO2 MS4FJ, which processes the order and can respond with the order id and cost.

Why Open Source?

As a small business, Sam’s resources are limited. Her best strategy for implementing the solution is open source, which offers lower startup costs and effort compared to the high licensing fee and complications involved with the commercial vendors.

Being open source also allows Sam to download, learn and evaluate the product without a high investment, thus minimizing her business risks. Depending on the results of her evaluations, he could go forward or ‘throw away’.

To grow in a competitive business  environment requires companies to differentiate. For small scale business  it becomes more of a challenge to implement such solution due to resource limitations. The seamless integrated capability provided by the open-source WSO2 Platform provides business a low risk and cost effective technology to build and deliver real-time business value to their clients.

The code for this use case

Listed below are what you need to recreate this discussion as a demo:

 

Pre-Requisites

Down the following products and set the port offsets to run the servers on the same server. WSO2 APIM runs on the default offset (0) while the WSO2 CEP offset is 4.

Products
WSO2 API Manager 2.0.0
WSO2 Complex Event Processor 4.2.0
WSO2 MSF4J (WSO2 MicroServices Framework for Java)
WSO2 App Cloud

For simplification purposes the inventory details are stored as tables of a MySQL database.

Execute MySQL database script db_script.mysql to create ‘Inventory’ Database and ‘Rewards’ and ‘orders’ table.

WSO2 MSF4J

  1. Execute the Kopi-service’ java microservice
    1. <WSO2_MSFJ_HOME>/kopi-service/target / Java -jar kopi-service-0.1.jar

Alternatively the java microservice can be deployed in the WSO2 App Cloud.

WSO2 CEP Setup

  1. Setup email configuration for the output event publisher
  2. Copy the JDBC driver JAR file for your database to <CEP_HOME>/repository/components/lib.
  3. Startup the server
  4. Configure a data source as “CEP-DS”.  Select Mysql as the RDBMS and set the database as ‘Inventory’ created.
  5. The created datasource is referenced when defining ‘Event Tables’ when creating the Siddhi queries.
  6. Deploy “Streaming-CApp” CApp . The correct deployment should visualize an event flow as depicted.

WSO2 API Manager Setup

  1. Configure WSO2 API Manager to pass end user attributes as JWT Token.
  2. Copy the custom data publisher implementation (org.wso2.api.publish-1.0-SNAPSHOT.jar ) library to $API_MGR_HOME /repository/components/lib
  3. Startup the Server.
  4. Login to the API Publisher:
  5. Create and publish an  API with the following details
    1. Context -<context>
    2. Version – <version>
    3. API Definition
      1. GET – /order/{orderId}
      2. POST -/order
    4. Set HTTP Endpoint: http://<server-ip:port>/WSO2KopiOutletPlatform/services/w_s_o2_kopi_outlet_service
    5. Change the default API call request flow by enabling message mediation and uploading file  datapublisher.xml as the ‘In Custom Sequence’.
  6. Login to the API Store and subscribe to the created API
  7. Invoke API with an order exceeding available quantity { “Order”:{ “drinkName”:”Doppio”, “additions”:”cream”, “orderQuantity”:2000 } }

 

Predicting re-order levels

The re-order quantity is initially calculated based on a ‘re-order factor(ROF) and order quantity formula (ROF * order quantity). Siddhi provides a machine learning extension for predictive analytics. The reorder quantity can be predicted using machine learning model.  

The re-order data points calculated previously (with the formula) can be used as data sets to generate a machine learning model with WSO2 Machine Learner. A predicted re-order quantity is calculated based on the “Linear Regression” algorithm, with the “Reorder factor (ROF) and coffee type  as the features.

The siddhi query for predicting reorder quantity is commented under ‘Predict reorder quantity using Machine Learning extensions. It can be executed by replacing the query under ‘Calculating reorder quantity’.

Appendix: code

Custom Sequence

<?xml version=”1.0″ encoding=”UTF-8″?>

<sequence name=”publish-endUser” trace=”disable” xmlns=”http://ws.apache.org/ns/synapse”>

 <log level=”full”/>

 <property expression=”get-property(‘$axis2:HTTP_METHOD’)” name=”VERB”

   scope=”default” type=”STRING” xmlns:ns=”http://org.apache.synapse/xsd”/>

 <property expression=”get-property(‘transport’,’X-JWT-Assertion’)”

   name=”authheader” scope=”default” type=”STRING” xmlns:ns=”http://org.apache.synapse/xsd”/>

 <log level=”custom”>

   <property expression=”base64Decode(get-property(‘authheader’))”

     name=”LOG_AUTHHEADER” xmlns:ns=”http://org.apache.synapse/xsd”/>

 </log>

 <property expression=”base64Decode(get-property(‘authheader’))”

   name=”decode_auth” scope=”default” type=”STRING” xmlns:ns=”http://org.apache.synapse/xsd”/>

 <script description=”” language=”js”><![CDATA[var jsonStr= mc.getProperty(‘decode_auth’);

var val= new Array();

val=jsonStr.split(“}”);

var decoded= new Array();

decoded= val[1].split(“enduser\”\:”);

var temp_str= new Array();

temp_str=decoded[1].split(‘\”‘);

mc.setProperty(“end_user”,temp_str[1]);]]></script>

 <property expression=”get-property(‘end_user’)” name=”endUser”

   scope=”default” type=”STRING”/>

 <log level=”custom”>

   <property expression=”get-property(‘endUser’)” name=”Log_Enduser”/>

 </log>

 <class name=”org.wso2.api.publish.PublishMediate”>

   <property name=”dasPort” value=”7619″/>

   <property name=”dasUsername” value=”admin”/>

   <property name=”dasPassword” value=”admin”/>

   <property name=”dasHost” value=”localhost”/>

   <property name=”streamName” value=”Data_Stream:1.0.0″/>

 </class>

</sequence>

Siddhi Query

/* Enter a unique ExecutionPlan */

@Plan:name(‘Predict’)

/* Enter a unique description for ExecutionPlan */

— @Plan:description(‘ExecutionPlan’)

/* define streams/tables and write queries here … */

@Import(‘API_Stream:1.0.0’)

define stream APIStream (drinkName string, additions string, orderQuantity double, endUser string);

@Export(‘allOrder_Stream:1.0.0’)

define stream allOrderstream (drinkName string, qtyAvl double, qtyPredict double);

@Export(‘predictStream:1.0.0’)

define stream predictStream (drinkName string, qtyPredict double);

@Export(‘Order_Stream:1.0.0’)

define stream orderStream (drinkName string, orderQty double, qtyAvl double, qtyOrder double, ROF double); 

@Export(‘reOrder_Stream:1.0.0’)

define stream reOrderStream (drinkName string, qtyAvl double, qtyPredict double);

@Export(‘outOrder_Stream:1.0.0’)

define stream outOrderStream (drinkName string, qtyOrder double, qtyReorder double, ROF double);

@Export(‘ULPointStream:1.0.0’)

define stream ULPointStream (subScriber string, points double);

@Export(‘totPointStream:1.0.0’)

define stream totPointStream (subScriber string, totPoints double);

@Export(‘FreeOrderStream:1.0.0’)

define stream FreeOrderStream (subScriber string, points double); 

@from(eventtable=’rdbms’, datasource.name=’CEP-DS’, table.name=’orders’)

define table drinkEventTable(drinkName string, qtyAvl double, qtyOrder double, ROF double);

@from(eventtable=’rdbms’, datasource.name=’CEP-DS’, table.name=’rewards’)

define table pointEventTable(subscriber string, points double);

from APIStream#window.length(0)as t join drinkEventTable as d

on t.drinkName==d.drinkName

select t.drinkName as drinkName, t.orderQuantity as orderQty, d.qtyAvl as qtyAvl,d.qtyOrder as qtyOrder, d.ROF as ROF

insert into orderStream; 

/* ——Drink Reordering————- */

/* —–Calculating reorder quantity———– */ 

from orderStream#window.length(0) as p join drinkEventTable as o

on o.drinkName==p.drinkName

select o.drinkName,o.qtyAvl,(p.orderQty* p.ROF) as qtyPredict

insert into allOrderstream;

/*———————Predict reorder quantity using Machine Learning extentions—————*/

/*

from orderStream

select drinkName,ROF

insert into ROF_Incoming;

from ROF_Incoming#ml:predict(‘registry://_system/governance/ml/Reorder.Model’,’double’,drinkName,ROF)

select drinkName, qtyReorder as qtyPredict

insert into predictStream;

from predictStream#window.length(0) as p join drinkEventTable as o

on o.drinkName==p.drinkName

select o.drinkName,o.qtyAvl, p.qtyPredict

insert into allOrderstream;

*/

/*——————————————————–*/

partition with (drinkName of allOrderstream)

begin @intro(‘query3’)

from allOrderstream[qtyPredict>=qtyAvl]

select drinkName,qtyAvl,qtyPredict

insert into #tempStream2;

from e2=#tempStream2

select e2.drinkName, e2.qtyAvl,e2.qtyPredict

insert into reOrderStream

end;

from orderStream[(qtyAvl-orderQty)>=0]#window.length(0)as t join drinkEventTable as d

on t.drinkName==d.drinkName

select t.drinkName as drinkName,(d.qtyAvl – t.orderQty) as qtyAvl

update drinkEventTable

on drinkName==drinkEventTable.drinkName; 

/*——————————————– */

 /*—– Offer free drink ——-*/

from APIStream

select endUser as subScriber ,orderQuantity as points

insert into ULPointStream;

 from ULPointStream as u join pointEventTable as p

on u.subScriber == p.subscriber

select u.subScriber as subscriber ,(u.points+p.points) as points

update pointEventTable

on subscriber==pointEventTable.subscriber;

from ULPointStream[not(pointEventTable.subscriber==subScriber in pointEventTable)]

select subScriber as subscriber,points

insert into pointEventTable;

from ULPointStream as u join pointEventTable as p

on u.subScriber == p.subscriber

select u.subScriber as subScriber,p.points as totPoints

insert into totPointStream;

 partition with (subScriber of totPointStream)

begin @info(name = ‘query4’)

from totPointStream[totPoints>=100]

select *

insert into #tempStream;

 from e1= #tempStream

select subScriber, totPoints as points

insert into FreeOrderStream

end ;

/*————————————*/

Better Transport for a better London: How We Won TfL’s Data in Motion Hackathon

Transport for London (TfL)  is a fascinating organization. The iconic red circle is practically part and parcel of the everyday life of the 1.3 billion people that the TfL network transports across London.

As part of their mandate, TfL is constantly on the search for ways better manage traffic, train capacity, maintenance, and even account for air quality during commutes. These are some very interesting challenges, so when TfL, Amazon Web Services and Geovation hosted a public hackathon, we at WSO2 decided to come up with our own answers to some of these problems.

Framing the problem

29413875894_f7ba6582b0_k
TfL’s Chief Technical Architect, Gordon Watson, catches up with the WSO2 team. Photo by TFL.

TfL pushes out a lot of data regarding the many factors that affect public transport within Greater London; a lot of this is easily accessible via the TfL Unified API from https://api.tfl.gov.uk/. In addition to volumes of historical data, TfL also controls a network of SCOOT traffic sensors deployed across London. Given a two-day timeframe, we narrowed our focus down to three main areas:

  1. To use historical data regarding the number of passengers at stations to predict how many people would be on a selected train or inside a selected station
  2. To use Google Maps and combine that with sensor data from TfL sensors across the city to pick the best routes from point A to B, while predicting traffic, five to ten minutes into the future, so that commuters could pick the best routes
  3. To pair air quality data from any given region and suggest safer walking and cycling routes for the denizens of Greater London

Using WSO2 Complex Event Processor (which holds our Siddhi CEP engine) with Apache Spark and Lucene (courtesy of WSO2 Data Analytics Server), we were able to use TfL’s data to build a demo app that provided a solution for these three scenarios.

1

For starters, here’s how we addressed the first problem. With data analysis, it’s not just possible to estimate how many people are inside a station; we can break this down to understand traffic from entrance to a platform, from a platform to the exit, and between platforms. This makes it possible to predict incoming and outgoing crowd numbers. The map-based user interface that you see above allows us to represent this analysis.

The second solution makes use of the sensor network we spoke of earlier. Here’s how TfL sees traffic.

2

The red dots are junctions; yellow dots are sensors; dashed lines indicate traffic flow. The redder the dashed lines are, the denser the traffic at that area. We can overlay the map with reported incidents and ongoing roadworks, as seen in the screenshot below:

3Once this picture is complete, we have the data needed to account for road and traffic conditions while finding optimal routes.

This is what Google suggests:

4

We can push the data we have to WSO2 CEP, which runs streaming queries to perform flow, traffic, and density analytics. Random Forest classification enables us to use this data to build a machine learning model for predicting traffic – a model which, even with relatively little data, was 88% accurate in our tests.  Combining all of this gives us a richer traffic analysis picture altogether.

5

For the third problem – the question of presenting safer walking and cycling routes using air quality – our app pulled air pollution data from TfL’s Unified API.

This helps us to map walking routes; since we know where the bike stations are, it also lets us map safer cycling routes. It also allows us to push weather forecasts and air quality updates to commuters.

A better understanding of London traffic

In each scenario, we were also able to pinpoint ways of expanding on, or improving what we hacked together. What this essentially means is that we can better understand traffic inside train stations, both for TfL and for commuters. We can use image processing and WiFi connections to better gauge the number of people inside each compartment; we can show occupancy numbers in real-time across screens in each station, and on apps, and assist passengers with finding the best platform to catch a less crowded compartment.

We can even feed Oyster Card tap data into WSO2 Data Analytics Server, apply machine learning to build a predictive model, and use WSO2 CEP to predict source to destination travel times. Depending on screen real estate, both air quality and noise level measures could be integrated to keep commuters better informed of their travelling conditions.

How can we improve on traffic prediction? By examining historical data, making a traffic prediction, then comparing that with actual traffic levels, we could potentially predict  traffic incidents that our sensors might have missed. We could also add location-based alerts pushed out the commuters – and congestion warnings and time-to-target countdowns on public buses.

We have to say that there were a number of other companies hacking away on excellent solutions of their own; it was rather gratifying to be picked as the winners of the hackathon. For more information, and to learn about the solutions that we competed against, please read TfL’s blog post on the hackathon.

WSO2 API Manager 2.0.0: Better Statistics

Introduction

Any API management tool requires the ability to view statistics – accurately and with details essential to the operation of APIs. WSO2 API Manager 2.0.0 introduces powerful analytics features along these lines by using the capabilities of WSO2 Data Analytics Server (WSO2 DAS), bringing all API, application and subscription related statistics to one location.

image05

New additions

In order to help you make sense of these analytics, we’ve added the following graphs:

1.Published APIs Over Time

This graph can be used to check API creation rates. A user can check APIs based on the API provider or can view statistics related to all API providers.

image05

2. API Latency

The API Latency breakdown graph can be used to check time consumption for each level in the message flow. It shows total time taken to handle a request and provides the time each component spends on that request – for example, a user can check the amount of time spent on the authentication process and so on. This graph is useful for identifying delays in the API.

image013. API Usage Across Geo Locations

This graph provides statistics based on the location of the API request. It uses X-Forwarded-For header to identify the location of the user.

image06

4. API Usage Across Usage Agent

This graph provides information related to the access mechanism of the API, and helps provide insight into the kind of devices and platforms used to access the API – mobiles, computers and so on.

image08

We’ve also added an Applications Created Over Time graph under the Applications section. As with the API creation graph, this provides statistics related to application creation. Users can drill down the data based on the registered user and the API in question.

image02

In addition to these come the subscription graphs: Developer Signups Over Time and Subscriptions Created Over Time, both of  which are self-explanatory.

image00

image07

How These Statistics Work

WSO2 Analytics Server for API Manager is essentially a fully functional WSO2 DAS server.

image03

In a nutshell, WSO2 API Manager 2.0.0 sends the following event streams to API Manager Analytics:

org.wso2.apimgt.statistics.request:1.1.0

org.wso2.apimgt.statistics.response:1.1.0

org.wso2.apimgt.statistics.fault:1.0.0

org.wso2.apimgt.statistics.throttle:1.0.0

org.wso2.apimgt.statistics.workflow:1.0.0

org.wso2.apimgt.statistics.execution.time:1.0.0

org.wso2.analytics.apim.alertStakeholderInfo:1.0.0
The APIM Analytics Server process these events and writes the summarized data to a shared database. WSO2 API Manager then queries statistics from this database to display in the API Manager Publisher and Store.

Footnotes

One significant change in WSO2 API Manager 2.0.0 statistics is the removal of the graphical user interface (GUI) for configuring statistics. In WSO2 API Manager 1.10, a GUI was provided to handle this DAS-related configuration. We’ve moved this functionality back into the api-manager.xml file.

We’ve also taken out the DAS Rest client – with 2.0.0, the default and only implementation is APIUsageStatisticsRdbmsClientImpl, which uses an RDBMS client to get data from the database.

Since this is essentially a DAS server, you can use features such as gadgets and realtime analytics (using Siddhi) for better or more complex visualizations of the events sent from WSO2 API Manager.

Eurecat: using iBeacons, WSO2 and IoT for a better shopping experience

Eurecat, based in Catalonia, Spain, is in the business of providing technology. A multinational team of researchers and technologists spread their efforts into technology services, consulting and R&D across sectors ranging from Agriculture to Textiles to the Aerospace industry. By default, this requires them to work in the space of Big Data, cloud services, mobile and the Internet of Things.

One of their projects happened to involve iBeacons in a store. In addition to transmitting messages, the low-energy, cross-platform Bluetooth BLE-based sensors can detect the distance between a potential user and themselves – and transmit this information as ‘frames’. Using this functionality, a customer walking outside the store would be detected and contacted via an automated message.

image00

Upon arriving at the entrance to the store, the customer would be detected by beacons at the front of the shop (near) and at the back of the shop (far). This event itself would be a trigger for the system – perhaps a notification for a store clerk to attend to the customer who just walked in. The possibilities aren’t limited to these use cases: with the combination of different positions and detection patterns, many other events can be triggered or messages pushed.

To implement this, Eurecat architected the system thusly.

image01

The process is set in motion by the iBeacon, which keeps broadcasting frames. These are picked up by the smartphone, which contacts the business services. Complex Event processing would occur here to sort through all these low-level events in real-time. The bus then funnels this data to where it needs to go – notification services, third parties, interfaces and databases.

The WSO2 Complex Event Processor (CEP) and the WSO2 Enterprise Service Bus (ESB) fit in readily, with the ESB collecting the events and passing them on to the processing layer.

image02

Jordi Roda of Eurecat, speaking at WSO2Con EU 2015, detailed why they choose to go with WSO2: the real-time processing capabilities of CEP, the array of protocols and data formats it can handle, and the Siddhi language, which enabled them to easily construct the queries that would sift through the events. The ESB, said Jordi, they selected because of its performance, security and connectivity it offered.

At the time of speaking, Eurecat had improvements pending: data analytics, a wifi-based location service, better security and scalability.

image03

At WSO2, we’re delighted to be a part of Eurecat’s success – and if your project leads you along similar paths, we’d like to hear from you. Contact us.[a] If you’d like to try us out before you talk to us, our products are 100% free and open source – click here to explore the WSO2 Enterprise Service Bus or here to visit the WSO2 Complex Event Processor.

Capgemini, WSO2 and the new UN ecosystem

Ibrahim Khalili is a system integration analyst at Capgemini, a multinational that’s one of the world’s foremost providers of management consulting, technology and outsourcing. Headquartered in Paris, Capgemini has been running since 1967, and now makes over 11 billion EUR in revenue.

ibrahim

Capgemini and WSO2 have a history of working together. One of Capgemini’s recent projects was for the United Nations – to build a new reference architecture for UN agencies to function across a connected technology platform. Khalili, speaking at WSO2Con Asia 2016 in Colombo, Sri Lanka, outlined the three major goals of the new platform.

Whatever they designed had to allow beneficiaries, donors, citizens and the UN’s increasingly mobile workforce to access the functionality and information of agencies regardless at “anywhere, anytime, on any device”; it had to handle information, people and devices in a much smarter and more cost-efficient way that the UN was doing already. It also had to break out the data and bring the UN’s agencies into the world of an API ecosystem.

To put this into finer context, we’re talking about a system that can handle assets, finances, information and humans across a diverse array of agencies – including the nitty gritty of fundraising, running initiatives, and reporting that are key to most UN operations. What they required was what Khalili calls a “platform enabled agency” – more or less a complete update to operational infrastructure, with APIs exposing services, information, and functionality across the board.

Their solution starts with an integration layer that connects to all legacy systems, providing a view of all the data that can be managed. On top of that goes the process layer, which contains the functionality, and on top an API layer exposing the platform’s services and data.

capgemini-abstract

Once the logical framework was done, Capgemini started filling it in. At the very bottom go IaaS services like VMWare. On top of that comes an ERP universe of sorts – functionality from SugarCRM, Talend, WSO2 Application Server, WSO2 Complex Event Processor, and others, connected by the WSO2 ESB. WSO2 Enterprise Mobility Manager, WSO2 API Manager, and WSO2 User Engagement Server face outwards, allowing this functionality to be used. WSO2 Identity Server wraps around the entire platform, handling ID and authentication.

 

That gives Capgemini – and the UN – not only a cleaner, layered architecture, but one that brings in better scalability as well as a Devops approach. But above all, the chief advantage, says Khalili, is that it’s also open source. With WSO2 products, Capgemini has complete freedom to customize, take apart or rebuild whatever’s required to make a better platform. There’s no stopping innovation.

Capgemini’s not the only one who can leverage our technology. All WSO2 products are free and open source.

Go to http://wso2.com/products/ to download and use any part of our middleware platform. For more information on Capgemini’s solution for the UN, watch Ibrahim Khalili’s full presentation at WSO2Con here.