Category Archives: Featured

Motorola Mobility: Using WSO2 Integration Platform to Increase Business Agility

Companies all over the globe are realizing the power of lean technology on the cloud and Motorola Mobility is one of them that’s taking action towards wielding this power. In February 2017, Sri Harsha Pulleti, an integration architect at Motorola Mobility and Richard Striedl, an advisory IT architect at Motorola Mobility, spoke at WSO2Con USA 2017 about their move to a hybrid cloud and container architecture with zero-touch automation.

A few years ago, on the day after thanksgiving, Motorola’s website crashed, resulting in the loss of many transactions from buyers who were flooding in to get their discounts. That’s when they started questioning how it happened, why it happened, and what they could do about it, explained Sri. All their web services were running through heavy-weight enterprise service buses (ESBs) in their data centers that didn’t have any other technical capability. They needed to move away from this to a lightweight platform in the cloud.

After evaluating many vendors they found WSO2 and its lightweight ESB – just what they had been looking for. Sri explained that they could quickly spin up instances of it and even set auto-healing and auto-scaling capabilities. WSO2 ESB (now extended as WSO2 Enterprise Integrator, which includes all the other key products and technologies from the WSO2 Integration Platform) also supports Amazon Web Services (AWS), which was their first option for cloud computing services. After choosing their vendor, Motorola began to make the necessary changes in their environment by re-architecting the system, setting up multiple ESBs and moving to a micro-platform architecture.

A year later, thanksgiving came along and this time everything went smoothly. “It was perfect, there were no issues and everything was absolutely fine”, explained Sri. However, a few months later, they realized that this was costly. Sri was given the challenge of finding something with the same capabilities as AWS, but at a lower cost. That’s when they started looking at OpenStack: an open source software for creating private and public clouds. It created an environment with similar capabilities to AWS and allowed them to set up their own data centers. After discussing further, they decided to run both environments (AWS and OpenStack) parallely and scale them up or down as needed.

This time, they decided to use containers, which allowed them to package their software into standardized units for development, shipment and deployment. But why? It’s lightweight, flexible and easy to scale. Sri then went on to discuss the importance of emphasizing collaboration and communication between developers as well as IT through DevOps: “It’s something everybody wants to achieve”. Instead of having just a DevOps team to achieve this, they made a zero touch automation DevOps platform. This homegrown application called Debug 360 built on open source products allows their developers to focus on developing the code and checking it into a repository while the end-to-end automation takes care of the rest. It now takes less than a week to complete any new development in an integration model.

Motorola now has WSO2 ESB on AWS and OpenStack, one without containers and one with. The next step will be to integrate these instances to achieve their ultimate goal of spinning up instances in both environments, Sri noted.

Motorola Mobility Advisory IT Architect Richard Striedl further explained the concept of cloud elasticity. He stated that they have learnt a lot especially in terms of enhancing DevOps while working with WSO2 the last few of years. The requirements for cloud elasticity included having the same DevOps procedures, cloud capabilities and application code and auto-scaling.

“We’re evaluating WSO2 API Manager,” said Richard while explaining their need for APIs to manage the environment, build the framework and have more control over it. At present, they have 35 applications with 90% of traffic going through OpenStack and 10% going through AWS. Richard concluded by exploring their future plans of dockerizing with data services and message brokering capabilities available in the new WSO2 Enterprise Integrator. “We might even take that step towards Ballerina as we all learned today,” he added.

To learn more about how Motorola Mobility is moving to the cloud through zero touch automation listen to Sri’s and Richard’s talk at WSO2Con USA 2017.

West Interactive: Using WSO2 Identity Server to Enhance Customer Experience

Headquartered in Omaha, West Corporation is all about telecommunication – be it conferencing solutions, safety services, interactive voice response solutions or speech application automation. Pranav Patel, the vice president of systems development at West Interactive, recently spoke at WSO2Con USA 2017 about the unique customer experience they offer through their multi-tenanted role-based identity and access management solution built using WSO2 Identity Server.

An increasing numbers of users today are turning to various different channels like the web, mobile devices, and social media to interact with vendors. Pranav explained that knowing the customer and making sure that they can access West Interactive’s services from whichever channel they prefer is a key requirement for them.

West has been in the telecommunication industry for the last 30 years, and quite commonly, have many solutions that are siloed and distributed. Connecting all these solutions was a major challenge they needed to overcome in order to provide a holistic experience to their customers, explained Pranav. This meant dealing with and managing various different identities that belonged to many different customer portals. They needed to create a solution that revolves around centralizing user identities to a single user portal and creating an efficient identity and access management system.

Pranav then examined the requirements they needed to meet in order to achieve operational efficiency, easily manage accounts, save costs, and provide great customer experience. Other than the evident single sign-on and federation requirements, multitenancy with hierarchical tenant management was an important feature that enabled them to serve all their tenants (a client of West represented as a domain in the system) and users (individuals that require access to the portal and are grouped at the tenant level) through their portal. The system also needed to enforce rule-based access control that allows access to certain products (web applications that need to be integrated) depending on who the user is. In addition to this, they had corporate policy requirements for passwords, needed to maintain password history and had a password expiry date that prompted users to frequently change the password. Audit logging and user bulk imports were some other requirements.

“WSO2 fulfilled several of our requirements out-of-the-box, especially support for various protocols and heterogeneous multiple user stores,” observed Pranav. He went on to explain that they could easily extend the product and customize it for any features that it didn’t already have, making it the perfect solution for West.

WSO2 Identity Server is used for

  • Introducing a relationship hierarchy between the parent tenant and child subtenant and allowing multi-tenancy
  • Asking for and storing answers to five security questions per user
  • Defining permissions or roles for products (web applications) and users
  • Providing single sign-on and federation for users
  • Allowing employees to mimic a user and see how they perceive the user portal
  • Enforcing password policies set by tenants

Pranav expressed how WSO2 Identity Server meets all their current requirements and how they would like to introduce customizable login pages (by tenant), two-factor and multi-factor authentication, automated user provisioning and self-registration among other features in the future. He concluded by saying they were looking forward to adding WSO2 Data Analytics Server to the mix in order to monitor what’s really going on in the system.

To learn more about West Interactive’s story listen to Pranav’s talk at WSO2Con USA 2017.

WSO2Con USA 2017 – ballerinas, blockchain, oxygen bars and more!

San Francisco met us with bitter, cold winds, but we didn’t let that stop us from hosting the best user conference ever! With a fully restructured agenda, major product and roadmap reveals and phenomenal entertainment, this year’s WSO2Con USA was bigger and better than ever.

This year, we even had an actual oxygen bar inside the WSO2 Oxygen Bar: a place where attendees were able to meet with WSO2’s solutions architecture and engineering teams to answer all their questions on integration, API management, analytics, identity and access management, and the Internet of Things.

The conference started off with a bang, literally. The Taiko drummers marched on to stage and gave a warm welcome to everyone in the crowd!

WSO2 Founder, CEO and Chief Architect Dr. Sanjiva Weerawarana then presented the repositioning of WSO2’s product strategy to focus on providing a platform that enables digital transformation through integration, API management, identity and access management, smart analytics, and the Internet of Things.

Thomas Squeo, the senior vice president of digital transformation and enterprise architecture at West Corporation, followed this with another keynote. He explored how to digitally disrupt from within your enterprise by empowering your employees who will in turn engage with customers, transform your products and optimize operations.

Next, cue the ballerinas…

…which led to WSO2’s big reveal: Ballerinalang

Sanjiva examined Ballerina in detail: the general purpose, concurrent and strongly typed programming language with both textual and graphical syntaxes, optimized for integration. Following this, Sameera Jayasoma, the associate director, architect and lead choreographer of Ballerina showcased a few demos of the language in use.

The first day then broke off into the individual tracks; integration, analytics and strategy; with introductory, advanced and hand-on sessions as well as customer talks from Motorola Mobility and State of Arizona.

The end of day one was met with smooth Jazz sounds of The San Francisco Metro Combo at our networking event where attendees got to mingle with their peers and WSO2 experts.

The second day commenced with a keynote by State of Arizona Chief Technology Officer Jason Simpson who examined their cloud-first strategy towards becoming a digital government. He spoke on the challenges of moving their legacy technology and systems to the cloud with low budgets to meet the increasing demands of their users and went on to explain how State of Arizona overcame them.

This was followed by an insightful customer panel on bridging IT and business in digital transformation moderated by our very own Vice President of Solutions Architecture Asanka Abeysinghe. The panel consisted of Jason Simpson, the CTO of State of Arizona, Sri Harsha Pulleti, an integration architect at Motorola Mobility and Naresh Sikha, the chief architect at StubHub.

The second day’s tracks consisted of technical sessions on integration, analytics, API management, IoT and Security by WSO2 experts and many customer talks and panels including:

Yet another day of learning and exchanging ideas came to an end. But that wasn’t it for day two! Right after the sessions we went into the carnival themed conference party where attendees got to mingle, play old-school arcade games like Street Fighter, take funky pictures at the photo booth and dance the night away. DJ Nikkie Matteo scratched some killer beats but the band, Pacific Soul, stole the show and made everyone cut loose, footloose and kick of their Sunday shoes!

Day three went straight into the tracks that included technical sessions in the areas of IoT, security and devops. It also had a track specially for partners and one specially by partners. The customer and partner talks on day three included:

  1. IoT in Airline Operations, Suresh Subasinghe, Principal Architect, United Airlines
  2. Multi-tenanted, Role-based Identity & Access Management Solution at West, Pranav Patel, VP, Systems Development, West Interactive
  3. 0-60 with WSO2: API Management and User Authentication and Authorization Automation, Ismail Seyfi, Lead Software Architect, iJET International and Matt Barnes, Automation and Software Engineer, iJET International
  4. Enhancing Customer Experience with WSO2 Identity Server, Manoj Thirutheri, Director, SaaS and Tools Engineering, Nutanix
  5. Providing a Pathway from Stovepipe Systems to a Secure SOA Enterprise, Neil Custer, Senior Enterprise Systems Engineer, Eagle TG
  6. Rise to the Challenge with WSO2 Identity Server and WSO2 API Manager, Stefan Smeets, Enterprise Architect & Unit Manager, RealDolmen
  7. Journey of Migration from Legacy ESB to Modern WSO2 ESB Platform, Michael Enos, Senior Director, Techsoup and Ratnavel Sundaramurthi, Integration Architect, Aspire Systems
  8. Integrating Systems for University of Exeter Using Zero and the WSO2 Platform, Jack A. Rider, CTO, Chakray

We even had a session through Skype on Managing Verifone’s New Payment Device “Carbon” with WSO2’s EMM by Ulrich Herberg, a senior Java architect at Verifone, who couldn’t be physically present at the venue!

For the unconference sessions, Sameera had to get back on stage to do more Ballerina demos for the eager crowd. They just couldn’t get enough of it!

The attendees gathered in the main hall once again to listen to the last few keynotes of the conference.

Catheryn Nicholson, an engineer, entrepreneur, mother, and former U.S. Naval Officer, who is also the Founder of BlockCypher gave the first engrossing keynote on blockchain’s digital disruption and why developers, startups, corporations, academic institutions, and governments are all examining what blockchain technology can solve.

After exploring the past and present of blockchain technology and how it may influence your business, Catheryn made quite an exciting reveal on the future of blockchain and cryptocurrency. A group of open source developers with Harry Potter pseudonyms are currently developing a protocol (called Mimblewimble), which is still largely theoretical but has a lot of potential to solve a number of the clunkiness issues that bitcoin has. She predicts that the project will come out this year. So make sure to keep an eye out for that!

WSO2 Vice President of Solutions Architecture Asanka Abeysinghe, gave the closing keynote on a pragmatic approach to digital transformation through iterative architecture. He spoke of his experience as a consultant and evangelist of digital transformation and examined how to overcome technical and non-technical barriers to your vision by thinking big and acting small.

Before the final adieu, we made sure to recognize customers who have been with us for the past ten (of our eleven) years. Sanjiva presented Ron Murphy from eBay, Jey Bala from Kaiser Permanente, Prakash Iyer from Trimble, and Concur (who wasn’t able to attend the conference) with a small token of appreciation for taking our first steps with us, and helping us get to where we are today.

Stay tuned for news on our next conference, and hope to see you there soon!

Introducing WSO2 Enterprise Integrator 6.0

WSO2 started out as a middleware company. Since then, we’ve realized – and championed the fact that our products enable not just technological infrastructure, but radically change how a company works.

All over the world, enterprises use our products to maximize revenue, create entirely new customer experiences and products, and interact with their employees in radically different ways. We call this digital transformation – the evolution of a company from one age to another, and our role in this has become more a technology partner than a simple software provider.

In this realization, we’ve announced WSO2 Enterprise Integrator (EI) 6.0. Enterprise Integrator brings together all of the products and technologies WSO2’s created for the enterprise integration domain – a single package of digital transformation tools closely connected together for ease of use.

When less is more

Those of you who are familiar with WSO2 products will know that we had more than 20 products across the entire middleware stack.

The rationale behind having such a wide array of products was to enable systems architects and developers to pick and choose the relevant bits that are required to build their solution architecture. These products were categorized into several broad areas such as integration, analytics, Internet of Things (IoT) and so on.

We realized that it was overwhelming for the architects and developers to figure out which products should be chosen. We also realized that digital transformation requires these products to be used in certain common patterns that mirrored five fields: Enterprise Integration, API Management, Internet of Things, Security and Smart Analytics.

In order to make things easier for everyone, we decided to match our offerings to how they’re used best. In Integration, this means we’ve combined the functionality of the WSO2 Enterprise Service Bus, Message Broker, Data Services Server and others; now, rather than including and setting up many many products to implement an enterprise integration solution you can simply download and run Enterprise Integrator 6 (EI 6.0).

What’s it got?

EI 6.0 contains service integration or service bus functionality. It has data integration, service, and app hosting, messaging, business processes, analytic and tooling. It also contains connectors which enable you to connect to external services and systems.



The package contains the following runtimes:

  1. Service Bus

Includes functionality from ESB, WSO2 Data Services Server (DSS) and WSO2 App Server (AS)

  1. Business Processes

Includes functionality of WSO2 Business Process Server (BPS).

  1. Message Broker

Includes the functionality of WSo2 Message Broker (MB). However, this is not to be used for purely message brokering solutions; this runtime is there for guaranteed delivery integration scenarios and Enterprise Integration Patterns (EIPs).

  1. Analytics

The analytics runtime for EI 6.0, useful for tracking performance, tracing mediation flows and more.

In order to provide a unified user experience, we’ve made some changes to the directory structure. This is what it looks like now:

The main runtime is the integrator or service bus runtime and all directories relevant to that runtime are at the top level.

This is very similar to the directory structure we use for other WSO2 products; the main difference is the WSO2 directory, under which the other runtimes are available.

Under the other runtimes, you find the same directory structure as the older releases of those products, as shown below.

One might ask why we’ve included multiple runtimes instead of putting everything in a single runtime. The reason for doing so is the separation of concerns. Short running, stateless integrations will be executed on the service bus runtime while long-running and possibly stateful integrations will be executed on the BPS runtime. We also have optional runtimes such as message broker and analytics which will be required only for certain integration scenarios and when analytics are required, respectively.

By leaving out unnecessary stuff, we can reduce the memory footprint and ensure that only what is required is loaded. In addition, when it comes to configuration files, only files related to a particular runtime will be available under the relevant runtime’s directory.

On the Management Console

There’s also been a change to the port that the management console uses. The 9443 servlet transport port is no longer accessible; we now use the 8243 HTTPS port. Integration services, web apps, data services and the management console are all accessible only on the passthrough transport port, which defaults to 8243.

Tooling

Eclipse-based tooling is available for the main integration and business process runtimes. For data integration, we recommend using the management console itself from the main integration runtime.


Why 6.0?

As the name implies, EI is an integration product. The most widely used product in the integration domain is the WSO2 Enterprise Service Bus (ESB), which in the industry is known to run billions of transactions per day. EI is in effect the evolution of WSO2 ESB 5.0, adding features coming from other products. Thus, it’s natural to dub this product 6.0 – the heart of it is still the same.

However, we’ve ensured that the user experience is largely similar to what it was in terms of the features of the previous generation of products.  The Carbon platform that underlies all of our products made it easy to achieve that goal.

Migration to EI 6.0

The migration cost from the older ESB, BPS, DSS and other related products to EI 6.0 is minimal. The same Synapse and Data Services languages, specifications and standards have been followed in EI 6.0. Minimal changes would be required for deploying automation scripts such as Puppet scripts -the directory structures are still very similar, and the configuration files haven’t changed.

Up Next: Enterprise Integrator 7.0

EI 6.0 is based on several languages – Synapse for mediation, BPMN and BPEL for business processes, DSS language for data integration.

A user who wants to implement an integration scenario involving mediation, business processes, and data integration has to learn several languages with different tooling. While it’s effective, we believe we can do better.

At WSO2Con 2017, we just unveiled Ballerina, an entirely new language for integration. EI 7.0 will be completely based on Ballerina – a single language and tooling experience. Now the integration developer can concentrate on the scenario, and implement it using a single language and tool with first level support for visual tooling using a sequence diagram paradigm to define integration scenarios.

However, 7.0 will come with a high migration cost. Customers who are already using WSO2 products in the integration domain can transition over to EI 6.0 – which we’ll be fully supporting – while planning on their 7.0 migration effort in the long term; the team will be working on tooling which will allow migration of major code to Ballerina.

WSO2 will continue to develop EI 6 and EI 7 in parallel. This means new features and fixes will be released as WUM updates and newer releases of the EI 6.0 family will be available over the next few years so that existing users are not forced to migrate to EI 7.0. This is analogous to how Tomcat continues to release 5.x, 6.x, 7.x and so on.


EI 6.0 is available for download at wso2.com/integration and on github.com/wso2/product-ei/releases. Try it out and let us know what you think – it’s entirely open source, so you can take a look under the hood if that takes your fancy. To report issues and make suggestions, head over to https://github.com/wso2/product-ei/issues.

Need more information? Looking to deploy WSO2 in an enterprise production environment? Contact us and we’ll get in touch with you.

 

How we handle security at WSO2

A Proactive Strategy for Security Management

Any decent software development organization generally has a well-defined set of policies and procedures for security management.

At WSO2, we – as in, the Platform Security Team – constantly collaborate with other product teams, customers and external security researchers to manage overall security of all WSO2 product. In this post, we’d like to talk about how we do this.


Part One: in the realm of code

code-944504_1280

I: Designing for security

The first stage of software design is the gathering of requirements. In open source software, we tend to use third-party code quite a bit – it’s how open source works: we stand on the shoulders of giants.
However, we can’t simply use what code we think is suitable.

The first check comes here. At WSO2, if we identify any kind of third-party code to be used, we need it to be first approved by the Engineering Management group, who are an internal group of seasoned architects who function at a directorial level. For us, security comes as a first priority, not as an afterthought.

The next set of checks come in the design phase. What are the communication protocols being used? How secure are they? Where is the data stored, and how? What endpoints are we exposing to the public? We go through a series of use cases to identify where this design can be broken, and work with the product design team to integrate our security concerns from the start.

II: Review, rinse, repeat

The next part is obvious: every developer is responsible for writing clean code [1, 2, 3].

Code written by each developer goes through a process of code quality reviewing overseen by members of the relevant product team and the Platform Security Team. When submitting the code for reviewing, the developer has to submit the static code analysis reports – generated using tools like FindSecBugs [4]. This is a mandatory security check in the reviewing process. Only upon fixing all issues spotted in the first pass is code is merged to the repository.

III: Testing with the automated grindhouse

At WSO2, we use Jenkins quite a lot for automating the build process. It builds individual components; it packages components together; it constantly builds and re-builds.

A large part of our security testing is integrated right into this process. Jenkins first performs the OWASP Dependency Check [5, 6], which analyzes the project dependencies and produces vulnerability reports. Even after the selection process in the first stage is complete, there can be some vulnerabilities that we haven’t spotted – especially if they’ve only been discovered extremely recently.

Next, Jenkins uses FindSecBugs as a plugin; during each automated build cycle, it checks individual components and generates vulnerability reports, which are in turn submitted to the security team for review.

Jenkins also uses the OWASP Zed Attack Proxy for dynamic code analysis [7, 8]. During the dynamic security analysis, the entire URL tree of the product is scanned and well-known attacks and exploits are automatically performed; the results are reported. These reports, too, are investigated by the respective product team as well as the Platform Security Team.

Once the testing is complete and a product is ready to be released, the respective product team has to receive security clearance from the Platform Security Team. If any known vulnerabilities are still listed in the reports, the product team has to justify to us the existence of the reported vulnerability – a pretty hard job.

We find that developers may write code following all the best security practices, but when the code is merged together, it might still open up a vulnerability because of how everything integrates together.


 Part Two: when humans happen

astronaut-and-robonaut-shake-hands

I: Preparing for the real world

There’s a saying: no battle plan survives contact with the customer. Although security standards and processes are followed to the letter, our products have to run in the real world.

One of the most important things is building awareness. We put together a set of deployment patterns, security recommendations, and best practices to be followed when deploying our products; we also conduct public webinars for making awareness in security related topics for WSO2 users, which are available at wso2.com/library/webinars.

II: Building internal Champions

Sometimes there is a gap between the product team and the security team, since the members of the security team might not be specialists of the product.

In order to bridge this gap, we’ve have someone we call the ‘Security Champion’ in each product team. The Security Champion of the product team is responsible for maintaining the safety of the product and conducting vulnerability assessments.

All Security Champions (from different product teams) directly work with the Platform Security Team and share knowledge and experiences with each other. They also share the knowledge of the Platform Security Team back with the members of the product teams.

III: Patching up 

When a vulnerability is detected in a product, patches are created for all the versions that the issue exists in. If the severity of the vulnerability is catastrophic, these patches will be released to all customers immediately. If the severity is not catastrophic, we aggregate all patches developed during the month and release the lot at the end of the month as a security bulletin.

When a patch is ready, it’s sent out through WSO2 Update Manager (WUM), added to wso2.com/security-patch-releases and publicly announced. Every version of any given product supported by WUM will receive the patches automatically. Note that unless the product is supported by WUM, security patches are publicly released only for the very latest version of the products.

Moving forward, we’ve started recording this in Documentation at docs.wso2.com/display/Security/Security+Advisories for the sake of preserving more patch information. This effort is still recent but will add up over time.

IV: Responding to Vulnerability Reports

Technology gets updated every day and there are always new vulnerabilities and exploits discovered. We welcome contributions from our user community, developers, and security researchers to reinforce our product security. Over the years, a great many people – both customers and from the community -have helped us make our products the best they can be.

When someone reports a vulnerability, we try to verify the issue and respond to the reporter. If the vulnerability is a true positive, the patching process begins.

Generally, we do ask that the reporter refrains from publicly disclosing the vulnerability until we’ve patched it – this is to prevent anyone who might be vulnerable from being targeted.

We’re always looking for ways to make this easier. For example, we’ve set up wso2.com/security to serve as an easy, central point for our community to report issues. As time goes on,


 

References

[1] OWASP Secure Coding Practices https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

[2] Oracle Secure Coding Guidelines for Java http://www.oracle.com/technetwork/java/seccodeguide-139067.html

[3] SANS Secure Coding Guidelines https://www.sans.org/course/secure-coding-java-jee-developing-defensible-applications

[4] Static Code Analysis for Java using FindBugs Plugin and Identifying Security Bugs with FindSecurityBugs Plugin
http://tharindue.blogspot.com/2016/06/static-code-analysis-for-java-using.html

[5] OWASP Dependency Check CLI – Analyzing Vulnerabilities in 3rd Party Libraries http://tharindue.blogspot.com/2016/10/owasp-dependency-check-cli-analyzing.html

[6] Checking vulnerabilities in 3rd party dependencies using OWASP Dependency-Check Plugin in Jenkins https://medium.com/@PrakhashS/checking-vulnerabilities-in-3rd-party-dependencies-using-owasp-dependency-check-plugin-in-jenkins-bedfe8de6ba8#.ipu0b8u4o

[7] Dynamic Scanning with OWASP ZAP for Identifying Security Threats https://medium.com/@PrakhashS/dynamic-scanning-with-owasp-zap-for-identifying-security-threats-complete-guide-52b3643eee04#.nyy1fwiok

[8] Automating the boring stuff in development using ZAP and Jenkins : Continuous Integration
https://medium.com/@PrakhashS/automating-the-boring-stuffs-using-zap-and-jenkins-continues-integration-d4461a6ace1a#.jtknrzajt

Better Transport for a better London: How We Won TfL’s Data in Motion Hackathon

Transport for London (TfL)  is a fascinating organization. The iconic red circle is practically part and parcel of the everyday life of the 1.3 billion people that the TfL network transports across London.

As part of their mandate, TfL is constantly on the search for ways better manage traffic, train capacity, maintenance, and even account for air quality during commutes. These are some very interesting challenges, so when TfL, Amazon Web Services and Geovation hosted a public hackathon, we at WSO2 decided to come up with our own answers to some of these problems.

Framing the problem

29413875894_f7ba6582b0_k
TfL’s Chief Technical Architect, Gordon Watson, catches up with the WSO2 team. Photo by TFL.

TfL pushes out a lot of data regarding the many factors that affect public transport within Greater London; a lot of this is easily accessible via the TfL Unified API from https://api.tfl.gov.uk/. In addition to volumes of historical data, TfL also controls a network of SCOOT traffic sensors deployed across London. Given a two-day timeframe, we narrowed our focus down to three main areas:

  1. To use historical data regarding the number of passengers at stations to predict how many people would be on a selected train or inside a selected station
  2. To use Google Maps and combine that with sensor data from TfL sensors across the city to pick the best routes from point A to B, while predicting traffic, five to ten minutes into the future, so that commuters could pick the best routes
  3. To pair air quality data from any given region and suggest safer walking and cycling routes for the denizens of Greater London

Using WSO2 Complex Event Processor (which holds our Siddhi CEP engine) with Apache Spark and Lucene (courtesy of WSO2 Data Analytics Server), we were able to use TfL’s data to build a demo app that provided a solution for these three scenarios.

1

For starters, here’s how we addressed the first problem. With data analysis, it’s not just possible to estimate how many people are inside a station; we can break this down to understand traffic from entrance to a platform, from a platform to the exit, and between platforms. This makes it possible to predict incoming and outgoing crowd numbers. The map-based user interface that you see above allows us to represent this analysis.

The second solution makes use of the sensor network we spoke of earlier. Here’s how TfL sees traffic.

2

The red dots are junctions; yellow dots are sensors; dashed lines indicate traffic flow. The redder the dashed lines are, the denser the traffic at that area. We can overlay the map with reported incidents and ongoing roadworks, as seen in the screenshot below:

3Once this picture is complete, we have the data needed to account for road and traffic conditions while finding optimal routes.

This is what Google suggests:

4

We can push the data we have to WSO2 CEP, which runs streaming queries to perform flow, traffic, and density analytics. Random Forest classification enables us to use this data to build a machine learning model for predicting traffic – a model which, even with relatively little data, was 88% accurate in our tests.  Combining all of this gives us a richer traffic analysis picture altogether.

5

For the third problem – the question of presenting safer walking and cycling routes using air quality – our app pulled air pollution data from TfL’s Unified API.

This helps us to map walking routes; since we know where the bike stations are, it also lets us map safer cycling routes. It also allows us to push weather forecasts and air quality updates to commuters.

A better understanding of London traffic

In each scenario, we were also able to pinpoint ways of expanding on, or improving what we hacked together. What this essentially means is that we can better understand traffic inside train stations, both for TfL and for commuters. We can use image processing and WiFi connections to better gauge the number of people inside each compartment; we can show occupancy numbers in real-time across screens in each station, and on apps, and assist passengers with finding the best platform to catch a less crowded compartment.

We can even feed Oyster Card tap data into WSO2 Data Analytics Server, apply machine learning to build a predictive model, and use WSO2 CEP to predict source to destination travel times. Depending on screen real estate, both air quality and noise level measures could be integrated to keep commuters better informed of their travelling conditions.

How can we improve on traffic prediction? By examining historical data, making a traffic prediction, then comparing that with actual traffic levels, we could potentially predict  traffic incidents that our sensors might have missed. We could also add location-based alerts pushed out the commuters – and congestion warnings and time-to-target countdowns on public buses.

We have to say that there were a number of other companies hacking away on excellent solutions of their own; it was rather gratifying to be picked as the winners of the hackathon. For more information, and to learn about the solutions that we competed against, please read TfL’s blog post on the hackathon.