Tag Archives: Data

Better Transport for a better London: How We Won TfL’s Data in Motion Hackathon

Transport for London (TfL)  is a fascinating organization. The iconic red circle is practically part and parcel of the everyday life of the 1.3 billion people that the TfL network transports across London.

As part of their mandate, TfL is constantly on the search for ways better manage traffic, train capacity, maintenance, and even account for air quality during commutes. These are some very interesting challenges, so when TfL, Amazon Web Services and Geovation hosted a public hackathon, we at WSO2 decided to come up with our own answers to some of these problems.

Framing the problem

29413875894_f7ba6582b0_k
TfL’s Chief Technical Architect, Gordon Watson, catches up with the WSO2 team. Photo by TFL.

TfL pushes out a lot of data regarding the many factors that affect public transport within Greater London; a lot of this is easily accessible via the TfL Unified API from https://api.tfl.gov.uk/. In addition to volumes of historical data, TfL also controls a network of SCOOT traffic sensors deployed across London. Given a two-day timeframe, we narrowed our focus down to three main areas:

  1. To use historical data regarding the number of passengers at stations to predict how many people would be on a selected train or inside a selected station
  2. To use Google Maps and combine that with sensor data from TfL sensors across the city to pick the best routes from point A to B, while predicting traffic, five to ten minutes into the future, so that commuters could pick the best routes
  3. To pair air quality data from any given region and suggest safer walking and cycling routes for the denizens of Greater London

Using WSO2 Complex Event Processor (which holds our Siddhi CEP engine) with Apache Spark and Lucene (courtesy of WSO2 Data Analytics Server), we were able to use TfL’s data to build a demo app that provided a solution for these three scenarios.

1

For starters, here’s how we addressed the first problem. With data analysis, it’s not just possible to estimate how many people are inside a station; we can break this down to understand traffic from entrance to a platform, from a platform to the exit, and between platforms. This makes it possible to predict incoming and outgoing crowd numbers. The map-based user interface that you see above allows us to represent this analysis.

The second solution makes use of the sensor network we spoke of earlier. Here’s how TfL sees traffic.

2

The red dots are junctions; yellow dots are sensors; dashed lines indicate traffic flow. The redder the dashed lines are, the denser the traffic at that area. We can overlay the map with reported incidents and ongoing roadworks, as seen in the screenshot below:

3Once this picture is complete, we have the data needed to account for road and traffic conditions while finding optimal routes.

This is what Google suggests:

4

We can push the data we have to WSO2 CEP, which runs streaming queries to perform flow, traffic, and density analytics. Random Forest classification enables us to use this data to build a machine learning model for predicting traffic – a model which, even with relatively little data, was 88% accurate in our tests.  Combining all of this gives us a richer traffic analysis picture altogether.

5

For the third problem – the question of presenting safer walking and cycling routes using air quality – our app pulled air pollution data from TfL’s Unified API.

This helps us to map walking routes; since we know where the bike stations are, it also lets us map safer cycling routes. It also allows us to push weather forecasts and air quality updates to commuters.

A better understanding of London traffic

In each scenario, we were also able to pinpoint ways of expanding on, or improving what we hacked together. What this essentially means is that we can better understand traffic inside train stations, both for TfL and for commuters. We can use image processing and WiFi connections to better gauge the number of people inside each compartment; we can show occupancy numbers in real-time across screens in each station, and on apps, and assist passengers with finding the best platform to catch a less crowded compartment.

We can even feed Oyster Card tap data into WSO2 Data Analytics Server, apply machine learning to build a predictive model, and use WSO2 CEP to predict source to destination travel times. Depending on screen real estate, both air quality and noise level measures could be integrated to keep commuters better informed of their travelling conditions.

How can we improve on traffic prediction? By examining historical data, making a traffic prediction, then comparing that with actual traffic levels, we could potentially predict  traffic incidents that our sensors might have missed. We could also add location-based alerts pushed out the commuters – and congestion warnings and time-to-target countdowns on public buses.

We have to say that there were a number of other companies hacking away on excellent solutions of their own; it was rather gratifying to be picked as the winners of the hackathon. For more information, and to learn about the solutions that we competed against, please read TfL’s blog post on the hackathon.

Transform Your Enterprise IT: Integrate and Automate

Most enterprises deal with a variety of common IT problems to which they would find quick fixes. One such example is the need to maintain five different usernames and passwords to login to five different systems. Another typical example is the closing of a sales deal – the sales department would conclude the deal and ensure the goods are delivered; this would be updated on the sales records, however, when the finance department reconciles invoices against sales at the end of the quarter, there might be mismatches because the invoicing process was missed.

area-review

To address these issues, most enterprises will use a combination of basic IT and collaboration software to manage day-to-day requirements. And over time, these requirements will change, prompting a slight shift in the enterprise’s IT landscape too. This may result in a situation where different teams within the organization will find the most efficient ways to carry out tasks and meet their IT requirements with the use of packaged software, possibly by building their own, or even subscribing to more SaaS-type offerings.

While this might temporarily fix specific problems, it will pose long-term challenges as such measures are often not pre-planned or do not follow a particular IT roadmap. The actual negative effects of individual teams working in silos would only be felt when the company starts to grow and the use of various systems increase as well. Eventually, the use of several systems that don’t talk to each other will cause operational issues and even hurt motivation among employees.

The recurrent problems with these multiple systems working in silos include extensive manual effort, errors, blame, rework, frustration, complaints, and the need to manage multiple passwords. These in turn result in inefficiencies.

To address these challenges, the enterprise needs an easy-to-implement, cost-effective solution. There’s no guarantee though that there would be a plug and play type of system or one that could be customized to meet the enterprise’s exact requirements. The enterprise would seek a unique, bespoke solution that would either mean they change the way they work with existing software or rethink the software itself.

The most viable option would be to integrate the systems (which, of course, have proven to be efficient to meet a specific requirement) used by different functions and then explore some sort of automation that will provide relief to employees.

WSO2’s highly-acclaimed open-source middleware platform has the capabilities that enable seamless integration of IT applications, thus streamlining day-to-day business activities of a given enterprise. This in turn will boost efficiency and integration across business functions and teams and improve overall productivity as well.

For instance, WSO2 Identity Server (WSO2 IS) can define an identification for a user in a particular organization, enabling him/her to log into multiple systems on-cloud or on-premise with a single username/password.

The enterprise too will benefit as WSO2 IS offers provisioning capabilities that allow your IT to register and auto-provision new employees across multiple systems as well as easily de-provision them when they leave the organization.

WSO2 Enterprise Service Bus can meet all your integration challenges with its capability to connect various systems that speak different languages. It also comes with a defined set of connectors to further support integration of systems, be it on the cloud or on-premise.

Once all of your systems have been integrated, you can leverage WSO2 Data Analytics Server (WSO2 DAS) to pull reports from different functions within your organization and automatically collate data that will translate to valuable information required to make business decisions. WSO2 DAS has in-built dashboard capabilities that will automatically create and publish dashboards on a real-time basis.

Moreover, all WSO2’s products are 100% open source, which gives enterprises the freedom of choice and empowers the business with limitless possibilities to expand.

Learn more about WSO2’s comprehensive and open platform for your connected enterprise.

For more details on how to establish friendly enterprise IT and get more love from your team, watch this talk by WSO2’s VP Operations, Shevan Goonetilleke.