All posts by Yudhanjaya Wijeratne

Have You Checked Out the WSO2 Machine Learner Yet?

WSO2 Machine Learner is the latest addition to the WSO2 product stack. Or, to be precise, one of the latest additions: Machine Learner only held the crown for a few weeks before WSO2 Data Analytics Server become the new kid on the block.

Nevertheless, Machine Learner is something we’re particularly proud of. As you no doubt guessed, it’s a product meant for predictive business analytics: by building models and running data through them, it allows one to quite literally peek into possible futures.

Why is this important? Well, machine learning is rapidly becoming the new frontier for businesses. We’ve gone from educated guesses to dedicated analytics: now we’re trying to predict the future.

“I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s?” – Hal Varian, Chief Economist at Google

Hal Varian expects statisticians to be the next sexy job because of the sheer amounts of data and intelligence that companies need to chew through these days.  At WSO2, we expect machine learning to be an integral part of what makes that job sexy. And useful.

Done right, machine learning can easily replace a small army of statisticians – especially when it comes to enterprise big data. The world isn’t (yet) at the level where we can recreate a Minority Report for businesses, but there’s a lot of work that’s gone into this space. If you don’t know the field, you should: we’d recommend this short and beautifully designed Visual Introduction to Machine Learning, courtesy of R2D3.

If you know the field, you can skip the above and go directly to what Nirmal Fernando has to say. Nirmal leads the team working on the project, and his blogpost on ML gives a quick, concise breakdown of what WSO2 Machine Learner is capable of. Part of the challenge of predictive analytics is presenting the data intelligently, something we’ve paid careful attention to with WSO2 ML.

After that, you should probably check out what Srinath Perera, our Vice President of Research, has to say: he’s written a post called “WSO2 Machine Learner: Why Would You Care?” on his personal blog. This talks about how ML is built, and the uses you can put it to.

As you can see, ML completes our analytics offerings – we now offer batch, real-time, interactive and predictive analytics, and all of these components tie into each other beautifully for creating end-to-end analytics solutions for your business.

Of course, we have plans for making Machine Learner more powerful: we plan to add neural networks, algorithmic recommendation, anomaly detection, and more – as the product evolves, we’ll keep pushing out updates.

You can jump in and check this all out for yourself, of course! Machine Learner is free and open source. The Getting Started guide in our documentation will tell you everything you need to know to get a Machine Learner instance off the ground. Supun Setunga’s cooked up a detailed post on how you can set up that very first predictive model.

Download Machine Learner here, and you’re good to go. Set up your models right and you can peek into the future without actually having to go at 88 miles an hour in an old DeLorean. What’s not to like?

PS: If you’d like to call for support, or suggest features, don’t hesitate to contact us. We welcome bug reports, too!

WSO2Con Insights: Experian Uses WSO2 to Uncover Credit Intelligence

Some call Experian a credit score checking service, but that would perhaps be an injustice: this company, which now counts some 17,000 people among its employees, is the credit information company. So deeply ingrained are they that in certain countries, it’s common to be told “Go talk to Experian” when you have a problem with your credit. Nor does it stop there. Experian’s products have long since expanded beyond credit and into everything from financial education to digital user analytics: it’s now a business with revenues in the billions of dollars.

Experian has a very interesting set of needs. Day in and day out, customers arrive at Experian looking not only for credit reports, but for financial advice. Experian, analyzing their spending patterns and the ripple effects of those, is in a position to tell customers what to buy, what cards to keep, how to handle their bank accounts and loans, and a myriad of other details. In his talk at WSO2Con EU 2015 Rafael Garcia-Navarro, Head of Analytics at Experian explained how shifting from huge volume/low speed batch data processing to small volume/high speed data execution, helped them get their big data into shape.

The problem of real-time

Given the nature of what they do, Experian needs a lot of intelligence and data analysis power. In the world of credit intelligence, everything is linked – from where a user votes to the loans they’ve taken to the smartphone plan that he or she is on. In the past, they would process vast amounts of data offline and use that to make analyses.

To this, Experian added a requirement: real-time operation – defined by them as systems that could take data from marketing channels, process and react with the required information under the average human reaction time of 200 milliseconds.

More specifically, they needed systems that detect patterns at very high speeds, passing data in such a way that as to enable the full machinery to deliver complete results in under 200 milliseconds.

This is where the WSO2 Complex Event Processor comes into the picture. Experian were working with some serious names in data analytics – like Google – and they began using the WSO2 CEP to analyze the customer data in real-time.

The first step, is taking log files from digital platforms at the user level – cookies, if you will – to develop batch prediction models which help them decide what to promote to different users. The next step was to move out of purely historical data. Experian developed a Java application that simulates Google data; this data streams into WSO2 CEP.

“What happens there is Siddhi is running the queries to identify the events that are relevant for further analysis, and driving that in into a Java-based platform,” said Garcia-Navarro “We take the latest events that we’ve identified from the streaming application, and we take those events to re-run the score with the latest information that is available to users, and re-optimizing that with MarketSwitch.”

The system would constantly re-examine their data, updating it and fine-tuning it with the latest information, and drive the final, optimised decision back for execution on the marketing platforms. The challenge? In order to keep the whole system’s operation under 200 milliseconds, this particular sub-system had to do all of this at a mere 50 milliseconds. That’s a staggeringly small amount of time.

After a pause, he added, “This 50 milliseconds has now been brought down to between 3 and 5 milliseconds.”

From code to credit

WSO2’s involvement began, ironically, not in the field of marketing analytics, but with analyzing credit risk. Experian had a product (now called PowerCurve) traditionally built for mainframes in the credit risk space; it allowed credit risk analysts to design business rules visually. They wanted to use this along with MarketSwitch to examine a user’s propensity to buy something.

After the initial QuickStart program, Experian’s internal integrator – they have a team set aside for this – took it to the rest of the company. Even within Experian’s ocean of established technology stacks and software, the WSO2 CEP made a splash big enough to be a critical product. The first implementation connected to WSO2 CEP through WSO2 ESB. Later iterations directly connected to the Siddi processing engine.

Experian likes the way WSO2 has worked for them on this.

“We explored all the typical suspects,” said Garcia-Navarro. “The CEP world is well known, and CEP for high-frequency trading had been in use for years. We explored all those commercial providers, but we chose WSO2 for three key reasons:
The first is because it’s open source. We believe that whenever possible we need to start embracing open source much more widely in business.
The second one is the depth of knowledge of the support provided. WSO2 takes a lot of pride in their support model; they claim – rightly – that they don’t have pre-engineers, but engineers who work on on the product providing the support needed for clients. And when you start working with them you see the depth of skills and expertise that they have. That’s a big plus for us.
The final one is the depth of offerings. CEP we’ve built the prototype for and implemented in house in our data centers and infrastructure. We’re starting to look into many aspects – the next one we’re looking into is ESB, but not the only one.“

Right now, Experian is pushing Complex Event Processor to the limits. Because of the nature of their business, they’re heavily interested in the next steps that we take with CEP and some of the new things we’re working on in the data and analytics space.

For more information on Experian’s work with WSO2, view Rafael’s presentation at WSO2Con EU 2015.

Everything You Need to Know About WSO2 ESB 4.9

We’ve just rolled out a new upgrade to the WSO2 ESB; version 4.9.

While 4.9 is not a major release, we feel that there’s enough of a leap in functionality that a bit of reading might be required. As Chanaka Fernando puts it, it brings over 100 improvements and 600+ bug fixes, including the addition of inbound endpoints to make the ESB the ultimate integration engine.

To help you out, our engineers have compiled a number of blog posts that will bring you up to speed on what you need to know.

  1. What’s new in the WSO2 ESB
    We’d recommend starting with Chanaka Fernando’s piece on what’s new, what’s improved and and how all of this ties in to make the 4.9 release important.
  2. Understanding the Failover Message Processor
    The Failover Message Processor is a nifty addition that helps guarantee message delivery when your usual message stores are down. Prabath’s Ariyarantha’s extremely detailed blogpost runs you through the theory of how this useful message delivery mechanism works and how to deploy it in your solution.
  3. How to get JMS Inbound Coordination running
    The concept of inbound endpoints now allows the WSO2 ESB to support many more transports in a multi-tenanted environment. Jagath Ariyarathne writes about using an inbound endpoint for coordinating inbound messages from the Java Message Service.
  4. Using worker / manager clusters without a load balancer
    Worker / manager cluster setups are quite common, especially when it comes to scaling applications. Ravindra Ranwala describes how to set up, configure and deploy such a cluster without using a load balancer.
    If you do have a worker / manager cluster set up, Ravindra’s next tutorial is about Message Processor Coordination support and its two main use cases.
  5. Everything MQTT
    MQTT and MQTT support is a topic in its own right. We’d recommend starting with Sriashalya Srivathsan’s concise introduction to MQTT, followed up with Elilmatha Sivanesan’s short post about how the  WSO2 ESB supports MQTT. Then it’s back to Sriashalya on configuring Axis2 MQTT.
    Following up with improvements in the RabbitMQ space (listed on Maheeka Jayasuriya’s blogpost), we recommend Ravindra Ranwala’s writeup on the technical wizardry of getting the WSO2 ESB to use RabbitMQ, both as a producer and as a consumer.
    Then of course, there’s Kafka. In Kafka Inbound use cases, Kathees Rajendram writes on various ways to use the WSO2 ESB’s Kafka support to consume messages from Kafka brokers.
  6. Last, but not least: on preserving HTTP Headers
    The previous version of our WSO2 ESB had limited support for preserving HTTP header fields, but with new ESB 4.9.0 introduces new tools for this task. See Prabath’s blog for a short list of what those properties are.

If you haven’t downloaded the latest WSO2 ESB already, you can start over here (remember – it’s free and open source). For more information, contact us and we’d be happy to help you out.

WSO2Con Insights: Transforming the Ordnance Survey with WSO2’s Open Source Enterprise Middleware Platform

The British Ordnance Survey officially began in 1791.  Unofficially, it began some years before, when King George II commissioned a military survey of the Scottish Highlands. The work never really stopped. Today, over two hundred years later, Ordnance Survey Ltd is Great Britain’s national mapping agency: a 100% publicly owned, government-run company that’s one of the world’s largest producers of maps. They’re in the Guiness Book for the largest Minecraft map ever made – an 83-billion block behemoth that boggles map-makers around the globe.

Over the centuries, Ordnance Survey produced and sold some of the finest paper maps in the business. However they soon had a problem: people didn’t buy maps anymore. People downloaded maps. People accessed maps on a website. They just weren’t huge fans of the print and CD maps that the Ordnance Survey produced day in and day out.

The fault in our maps

Of course, the OS had to evolve. To solve the problem, they realized they had to move beyond retail and into selling data maps. The OS has a database called MasterMap, which is possibly the largest geospatial database in the world; it contains maps accurate to a single centimeter, updated some 10,000 times a day, underpinning some £100 billion worth of business activity in the UK alone. The OS wanted to make this database accessible. They needed a way to sell this data over the Internet; they wanted delivery mechanisms and a strategy.

The OS had been introducing technology to the map-making process since the 70’s, but this challenge was different. Their maps had to be supremely accessible – web pages, mobile phones, even someone on the highway looking for the next gas station, should all have to be able to access Ordnance Survey data without a fuss. They needed a system that understood context – the user on the smartphone might be using an official OS app for it; the user on the highway might be using something installed by the manufacturer or dealer who sold them the car; all of them would have to be treated and billed accordingly.

To do all this, they needed a robust API management solution that could recognize context, run the request into a system and deliver an output very, very fast.

Which is where WSO2 came in.

The wheels in motion

Initially, the OS worked with WSO2’s competition – Apigee. Apigee had a good APIM system, but they weren’t as good as the WSO2 platform in connecting to everything else. The initial QuickStart program proved WSO2 could do everything they wanted: within just two weeks, the OS had a POC system on their hands using WSO2 Complex Event Processor, WSO2 Identity Server, WSO2 API Manager and WSO2 Business Analytics Monitor, connected to Magento, which the OS was using.

WSO2’s comprehensive platform made everything significantly easier for the Ordnance Survey. Since the products integrated perfectly with each other, they no longer needed to look at many different vendors for everything they needed.

“When evaluating the vendors, we were looking for flexibility, we were looking for a willingness to get involved, to share information – to be on our side, I guess – and we were looking for a rich resource. Not a single product vendor with a range of products that would meet our needs,” said Hillary Corney, of the Ordnance Survey at WSO2Con Eu 2015.

The Ordnance Survey also liked the fact that WSO2 provided open-source without a premium price. Because Magento was also open source, and they had complete access to the code of the WSO2 solution, the OS team could very tightly integrate the two via SAML and SCIM.

“WSO2 is an open source platform, which allowed us to experiment early and learn in-depth without going through a complex procurement process, because in the government we have to go adhere to the EU tender process. And it’s a rich suite of products, which gave us confidence and allowed us to meet whatever circumstance we came up against,” added Corney. “The value add is really the speed of development; the fact that it’s open-source allows us to integrate, to customize it and bend the source code to our requirements in a way that’s not possible with straightforward off-the-shelf software. So I think that’s the biggest value for us – the flexibility.”

Ordnance – and the fact that it’s public sector – brought with it its own set of insights for WSO2 – especially in how public institutions work and the moving parts involved. As the engineers at WSO2 got used to these processes, we developed ways to get the project rolling without inflating the price, and WSO2 delivered exactly as promised. The Ordnance Survey is now working on not one, but two API delivery production environments with WSO2 software.

“The WSO2 team were embedded in our trenches, and the overall impression was that they really knew their stuff. It was one of the fastest proof of concept builds we’ve ever had; at the end of two weeks we were able to demonstrate almost everything – from start to finish.”

For further information on Ordnance Survey’s open source journey see Hillary Corney’s presentation slides at WSO2Con EU 2015.

WSO2Con Insights: How Government of Moldova Efficiently Digitized Public Services

Iurie Turcanu, the executive director of the eGovernment Center in Moldova, describes his country as “small but ambitious”. Indeed, few would suspect that Moldova, which officially declared itself a republic after the dissolution of the Soviet Union, would be one of the few nations in the world to leap headfirst into e-Government.

“Two years ago our government set a primary objective for our nation – to digitize all public services. That was a real challenge, because we have to do this fast and with relatively small budgets.” said Turcanu at WSO2Con EU 2015, where he and Artur Reaboi, an enterprise architect at the e-Government Center of Moldova, explained how they implemented this national interoperability platform, which streamlines public services delivery, both for citizens and businesses, as well as optimizing internal governmental business processes.

Apparently unfazed, they began to build a platform with the help of the WSO2 middleware platform.

The Foundation for e-Transformation

The government as a platform, or, as the World Bank puts it, the “Governance eTransformation Project” was undertaken by Turcanu and his team at the e-Government Center of Moldova, under the purview of the State Chancellory. Of course, they hit a number of problems on the way.

The first was the way things worked:

  1. Lack of communication between authorities, or even subdivisions of the same authority.
  2. Financial obstacles – some organizations sell data or access to data
  3. Technological obstacles – lack of standards, incompatibility, lack of documentation

The second set of problems lay in the scope of the problem. This was not a simple system of websites: they needed to built a system that would provide a unified connection bus between the different arms of government – judiciary, taxation, customs and more – and the citizens, and this needed to be accomplished with an absolute minimum of cost, time and human resources.

They needed authentication solutions for the fundamental task of identifying the citizen. They needed to gather different data structures, technologies and weave a single central system in between all these moving parts. They needed electronic messaging to communicate information to citizens. They needed electronic payment systems. Finally, they needed hosting to have the whole structure running.

This was also a relatively central system, which meant it needed the one thing that would be devastating to get wrong: scalability. Indeed, as Reaboi, explains, this was one of the first doubts their sponsors had.

“The WSO2 solution answered all of our problems,” says Reaboi. “Our requirements were many and had to cover many contexts…and future needs. This is why we use lots of WSO2 products; in the middle of this is the WSO2 Enterprise Service Bus.”

The end product is highly flexible. A public agency can come to them and be given the relevant interfaces to access the data they needed; where it came from or what format it was originally saved as, would not matter to the agency – the business of transforming the data is handled by the WSO2 Data Services Server and the ESB, which allowed them to open up data locked in diverse legacy systems without having to modify the existing data or software.

The WSO2 stack’s inherent multi-tenancy and performance let them scale this model quickly. Their test project, before it was announced to the public, took on some 5000 messages per second with only 2 ESB nodes. For 3.5 million people, they reasoned, that was more than enough scalability.

Today, much of Moldova’s government services are online. Everything from citizen IDs to payment mechanisms has been launched, driving not just digitization, but governmental reform towards a more efficient way of running a country. Indeed, as Turcanu put it, “In fact, it is the basis for public services engineering which is going on parallel to this reform. It acts as the core element of the public service engineering reform of Moldova.”

The e-Government Center hasn’t stopped. They’re now working on electronic visas, electronic procurement systems, a system for developing online registry and permit solutions – the idea is to digitize everything by 2020. Looking on their progress, WSO2 is proud to be right at the heart of Moldova’s trailblazing achievements.

To further understand Moldova’s eTransformation project see Artur Reaboi’s full talk at WSO2Con EU 2015.

Ten Years and Flying High!


Ten years ago, Sanjiva Weerawarana had a problem: he had an idea for a company, but not enough money.

An outsider would have been puzzled. Sanjiva technically didn’t need the money. A student of Royal College, he had moved to the USA for his PhD and started working for IBM in 1997. Technically, he had it made.

But he also had two more things: a massive passion for open source and a vision for a suite of middleware – an easily extensible software platform that let people build distributed applications. He also had stubborn, hard-nosed co-founder, Paul Fremantle, a product developer and author. They’d met over Paul’s book, somewhat creepily titled the ‘XML Files”.

So he quit IBM, setting up the framework for the business in Sri Lanka. Paul, wrapping up his work at IBM, joined shortly after. Some time later, developers began to apply to a new company called Serendib Systems. Behind closed doors, the father of XML, James Clark, said he’d invest, but the name had to change. Soon the founders came up with a new name. It was catchy, very geeky, and it had the wonderful effect of making students, developers and business people all over Sri Lanka scratch their heads in puzzlement.

“WSO2?” they would ask. “What does that even mean?”

Today, on the cusp of its tenth anniversary, WSO2 is a thriving business

Not only has it been in action for a decade; its products run businesses all over the world, connecting massive enterprises and billions of customers on a daily basis. The Sri Lankan company is now involved in everything from the takeoff and landing of Boeing aircraft to the e-Government platforms of European countries. There are table-tennis tables, carom boards, open kitchens and foosball tables in the lobby. The ’20 people in an office’ tagline has become irrelevant; now there’s just under 500 people in six offices, three of them in Sri Lanka. Nobody still knows what the name means.

The name WSO2 comes from (but does not stand for) Web Services (WS) Oxygen (O2). Web services are what created the opportunity for this company to enter and challenge the middleware market. The O, with a pulse in it, is because of how the business model works – it’s about giving support, and life support often involves oxygen (O2).

Of course, going from an office with 20 people to a multi-national company with hundreds of employees doesn’t happen overnight. The first employees were largely from the University of Moratuwa; some of them had made their mark at the Lanka Software Foundation, a nonprofit set up by Sanjiva in 2003. The first investors were James Clark, Paul Fremantle’s father and a mystery angel investor. The first product was WSO2 Tungsten; while it had nothing to do with the metal, it had everything to do with building applications on the Internet. Tungsten was followed by WSO2 Titanium, which would later become the famous WSO2 Enterprise Service Bus. Everything was open source – freely downloadable, easily modifiable, and completely contrary to the usual way of doing business in software.

By 2006, they needed more money. The fledgling finance team was ripping its collective hair out keeping things afloat. In May that year, Intel Capital invested 4 million dollars in WSO2, with perfect timing. Red Herring Asia called WSO2 one of the most promising private technology companies in Asia.

It’s almost tempting to say ‘and the rest is history’, but history generally records speeches, product reveals and profound statements made on stage; it tends to skip over days of coding and dedicated, passionate teams powering on through the nights. These were felt rather than written about. Challenges arose, were met and overcome. Awards kept coming in.

Gartner, the world’s most prestigious technology analyst firms, dubbed WSO2 a ‘visionary’ (and continues, to this day, to herald WSO2 as such). Developers found much to love in WSO2 software; it was, unlike others, completely open source, available to the community with no restrictions whatsoever.

In 2010, eBay decided to use WSO2 software. The world’s largest online marketplace tested WSO2 against a host of industry behemoths and decided that the Sri Lankan software outperformed everything else. By 2011 it was processing over $2,000 dollars per second and over a billion transactions a day. Today, it’s closer to 6 billion. eBay’s success validated WSO2: these Sri Lankan open source fanatics knew what they were doing. The company had a future.

Ten years later

WSO2’s products now run in a staggeringly diverse set of corporations – from international banks to airlines to automobile manufacturers to the makers of supercomputers. They’ve even become an integral part of the Apache Software Foundation’s open source community, pushing improvements, bug fixes, and updates to communities across the globe. Its competition is the likes of IBM, Oracle and RedHat, all heavyweights in the software industry. Its reach is international: every year, over 1000 trips are made across the world by WSO2 developers, helping global firms implement systems that impact millions of customers.

None of this is easy work. WSO2 succeeds partly due to a powerful, almost schizophrenic focus on the twin pillars of R&D and support. As it was all those years ago, competing with the juggernauts of the field still demands constant innovation and an absolute need to stay ahead of the curve. Along the journey, offices have shifted, headcounts have multiplied, email inboxes have burst, and the name ‘WSO2’ has become synonymous with success. The two products have become 24 interconnecting pieces of middleware. Every year, Fortune 500 companies and analysts look to WSO2 products to solve problems – ranging from creating cloud environments to interconnecting enterprise software that deals with millions of people every day.

As it celebrates its 10th anniversary, one thing’s clear: a bunch of developers from Sri Lanka can take on the world – and win.

It’s been one amazing decade.