Big Data and Politics: How the Internet sees the US Election

Nothing is a hotter topic than the US Election, especially if you’re a statistician at heart. Legions of us have been mesmerised by the idea of predicting who gets to be the most powerful President on the planet.

This year, however, it’s far more fun to kick back and watch the Internet collectively explode over each and every one of the candidates in the limelight. What with Clinton’s emailgate, Bernie’s economics, Ted Cruz’s household issues and Donald Trump’s existence …

WSO2 is a technology company. We looked around and realized that we had the tools to observe this theater on an unprecedented scale. We’d like you to join us.

Which is why we present to you the WSO2 Election Monitor.

At its heart, the Election Monitor is the WSO2 Enterprise Service Bus (ESB), Data Analytics Server (DAS) and Complex Event Processor (CEP). The ESB scans Twitter, pulling conversations about the US Election every second. DAS and CEP go to work on these tweets.

 The first thing we’ve done is build this (real-time) counter of the number of unique Twitter accounts talking about each camp. In a 24-hour time window, as of the time of writing, the Republicans seem to be dominating the Twittersphere.

image04

That’s a huge margin, isn’t it? Let’s find out why as we go along.

image01

This is firstly a live feed of what we’re getting from Twitter. The gray columns are the interesting ones: they display the most popular recent tweets – recent being not more than 12 hours ago. Donald Trump often dominates both fields. Occasionally, Bernie seems to break through. As of the time of writing, in the “Popular from candidates” column, Donald Trump has three tweets, one of them about a reporter touching him. The others are one tweet from Clinton Enough is enough”  and one from Bernie talking about deficits.

image00

This is consistent for what we’ve seen so far; ever since the site went live,  Trump’s snazzy one-liners have consistently gotten more retweets and favourites than Bernie and Clinton’s policy-centric tweets. It would appear that one man / tweep from the Republican party is more popular than every other candidate put together… are we really surprised that there’s more people talking about the Republicans than the Democrats?

But what about their followers? Using candidates’ hashtags, we can peek into the conversation by sifting through tweets and finding the most used conversations in that space.

Trump’s people are talking about the border. No surprise there. They’re also talking about New York. That corresponds with the fact that Hillary Clinton just took aim at Trump in a N.Y. ad. It shows a white Trump supporter sucker-punching an African American protester.  

Clinton? The email scandal hasn’t left her behind. There’s talk of war, probably because Clinton tweeted about defeating ISIS recently. There is a LOT of discussion regarding an upcoming debate with Bernie.

Bernie’s community, too, is talking about the debate. There’s few other clues in his wordcloud at the moment.

 Ted Cruz’s community is talking about his wife. That’s because he’s mired in a bit of controversy now: the family man is being dodgy about questions regarding his marriage. There’s a lot of questions about his principles.  

There’s one man missing from this: John Kasich. As of the time of writing, he’s got 143 votes. Cruz had 463. Trump has 736. They all need to hit 1,237 for nomination.

As remote as Kasich’s chances look in the polls, he barely exists on Twitter. For now, we must exclude him.

image02

Step three of the site is the community graph – or, as we call it, the attention graph. Here we map out the most popular accounts talking about the US election. The larger an account’s bubble is, the more popular it is.

What do we see? Donald Trump has gathered more attention to himself than any other tweep. It’s not even a small margin. Dan Scavino comes in at a distant second. Everyone else is miniscule, like little asteroids orbiting Planet Trump. And yet even those tiny accounts get over 2000 likes and retweets. These are the people who are essentially driving opinion on Twitter.

The fourth and final part is how the media’s opinion of a candidate changes over time. By analyzing news articles published online, we can determine shifts as campaigns unfold.

Consider how attitudes have changed towards Hillary. Here’s her standing on the 15th of March:

image05

Here’s her standing on the 17th:

image06

Opinion has swung her way. Examine the titles of the news articles on those days. On the 15th of March: “Was Hillary Clinton Bribed For Her Iraq War Vote?” And “The Cure to Hillary Clinton’s Problem With Millennials? Donald Trump.” Not that good.

On the 17th? “How Hillary Clinton Triumphed on Tuesday” and “Hillary Clinton Becomes Kween of Broad City”.  Short on the heels of a victory comes better press.

It’s fascinating to see how the American media react to candidates as they take on world events. Opinion on Trump, for example, hit rock bottom over his views on China and implications that supporters could go haywire.

Our collection of insights has just gotten started, of course. As the election unfolds, all of this will be running. While we can’t say that Internet is go along to predict who wins, we think it’s a pretty interesting gauge of what the people and the press of America are thinking.

Drop by https://wso2.com/election2016/. The project has been deprecated, but we’ve preserved a snapshot of the data so you can see what it was like.

Orchestration and Choreography – When To Use An ESB vs a Workflow Engine

Service chaining and workflows, commonly referred to as orchestrations, are common integration scenarios in enterprise systems development. There are two distinct type of orchestrations we deal with when realizing these systems:

  1. Short-running stateless orchestrations
    Service chaining messaging patterns that are more synchronous in nature and deals with transient data/sessions.
  2. Long-running stateful orchestrations
    Service chaining messaging patterns that are asynchronous in nature. These workflows take human input and approvals and are configured to run for longer durations with more persistent sessions.

If we look at these two types a bit closely the short-running ones are more straightforward; it mainly follows a request/response pattern, talks to multiple service endpoints, derives a response from one service call and composes the message to be sent to another service call. Enterprise integration patterns, such as message splitting, transformation, cloning and aggregation, are key building blocks in such orchestrations. The enterprise service bus (ESB) pattern fits this description quite well.

Screen Shot 2016-03-29 at 12

However, the long-running ones are more complex in nature and often involve human approval activities, work delegation, and assignments. A typical example for such a use case is a loan approval workflow; when a banking customer applies for a loan, the request triggers multiple services (1) credit approval service – this means connecting to multiple credit bureaus and valuing the customer’s credit worthiness, (2) evaluating the loan type (personal/home/vehicle, etc.), (3) finally, human approval (bank manager’s or loan department’s approval).

If you look closely, this particular workflow has different types of system interactions from a technical point of view. Not all interactions are long-running, yet the end-to-end scenario is so. The traditional architectural belief for such an end-to-end scenario is to model it with BPEL or BPMN (with a workflow/BPM tool), resulting in a complex model with human tasks and external links.
The common characteristic of this use case is that it is asynchronous in nature. The loan applicant will not get a response instantaneously; other parts, such as calling a credit bureau and getting the credit rating, is a synchronous stateless service call. Manager approval is a human interaction. Hence you do not have to follow the traditional path and build the entire workflow with the BPM tool. What you can follow is a hybrid model where you model the synchronous short-running stateless interactions with an ESB and long-running human interaction with a BPM tool. The proposed solution is depicted in the following architecture diagram.

Screen Shot 2016-03-29 at 12

The corresponding sequence of interactions has been illustrated in the following diagram.

Screen Shot 2016-03-29 at 12

WSO2Con Insights: How WSO2’s Open Source API Management Platform is Enabling BNY Mellon’s Digital Transformation

Let’s talk numbers. Bank of New York Mellon (BNY Mellon) runs a set of systems that track up to USD 30 trillion worth of wealth globally through investment management, investment services and wealth management. That’s about a quarter of all the world’s wealth of private assets, assets under management and assets under custody and/or administration.

When it comes to technology numbers, BNY Mellon operates a private cloud out of their own data centers, and has about 900 projects going on at any moment in time, run and managed smoothly by a 13,000 strong team.

During his talk at WSO2Con USA 2015, Michael Gardner, managing director and Head of the BNY Mellon Innovation Center explained how these numbers converged into creating the NEXEN digital ecosystem powered by WSO2’s API management platform, to transform the financial services industry.

The path of the open source code

Software driven disruption is impacting every company in every industry, Gardner noted, and the only way to survive is to keep moving in the same velocity, ability and agility as technology itself. Companies have evolved from mere ecommerce-related online retailers to managing entire customer relationships, to complete supply chain management and today, to digitized business operations. Such a company, according research firm Gartner, is defined as a ‘digital enterprise’.

Gardner noted that it’s critical for BNY Mellon to be a digital enterprise, to have the ability to accept new technologies and adapt, pushing very hard on it’s digital transformation and doing so by converging various technologies. He then went on to express why open source is now the center of their focus in this transformation.

“Open Source is very very important to us,” Gardner said. “We believe that open source is the future of enterprise collaboration. It’s not because it’s free. That’s great… but it (open source) becomes the basis for enterprises to collaborate together to evolve software mutually in ways that they need.”

The NEXEN digital ecosystem

BNY Mellon is bringing a collection of progressive software projects and technologies together, with an API program that enables the digital transformation of the organization to occur as an ecosystem.

Screen Shot 2016-03-25 at 12

“This transformation takes technology, it takes process and people, all these things working together,” Gardner comments. “It’s not easy to do this. If you are not driving that people part of it and the business process part of it, you are not going to accomplish the digital transformation.”

This convergence, Gardner said, was what finally lead BNY Mellon to create what is called the digital ecosystem of NEXEN. It involves BNY Mellon employees, covering both technology and business areas, customers as well as partner collaborators, including WSO2.

APIs – the critical link within the ecosystem

WSO2’s API management solution was chosen for the NEXEN ecosystem’s API program. “We selected WSO2 not just for the reason that it was open source. It gave us the chance to be able to actually work with the code, and understand the behavior of the system.”

How important are APIs for this digital transformation?

To keep the BNY Mellon cloud as modern as possible, the team constantly refactors backend systems. For this, smaller teams need to be empowered to carry out a given functionality.

“So APIs become really critical in being able to implement the most modern microservices based platform and architecture that we can,” Gardner noted. His team needs to ensure that whatever generation of technology a service is architected upon, that there is a modern REST API that’s available not only to interact with software systems, but to also allow people to consume these services.

“The microservices and architecture end up being the enabler of the digital transformation,” Gardner said. “If you’re going to be able to have the business move quickly, and adapt to new technology – you have to have APIs as the enabling lifeblood of it.”

Developer productivity too, according to Gardner was fundamental in achieving digital transformation. With 13000 people in technology at BNY Mellon, he explained how important it was to enable them to move at the same velocity as the technology itself, with modern API capabilities.

“At the end of the day what we are doing via the NEXEN program and the API Program is we are building a digital ecosystem that allows collaborations, and allows us to operate as a digital enterprise where every aspect of our business is digital.”

For more detailed information on BNY Mellon’s NEXEN API program, view Gardner’s WSO2Con USA 2015 presentation.

To understand more about BNY Mellon, check out ‘A History of BNY Mellon’ on Youtube.

WSO2Con Asia 2016 – highlights, pictures and turkey tweets!

Last week we concluded WSO2Con Asia 2016 with a bang! Over 250 attendees from across the region joined us, making it our biggest user conference to date.

We had inspiring keynotes, customer stories and technical sessions ranging from analytics and API management to cloud and integration. We also had lots of fun like twitter competitions and after parties.

Here’s a recap on some of our most memorable moments…

Day 1 was dedicated to tutorials. The sessions provided hands-on experience and deep dives into key WSO2 products. The ones on microservices and IoT were specially popular.

Tutorials

The official start of the conference however, was day 2 where the punchy sounds of Sachintha and the Beat Drummers kicked things off at the opening ceremony.

WSO2ConAsia-2016Opening

This was followed by the opening keynote by WSO2 Founder, CEO and Chief Architect Dr. Sanjiva Weerawarana. In his usual flair Sanjiva talked about WSO2’s progress in the past 10 years and how the WSO2 platform is being used by customers across industries such as  transport, government, entertainment, mobile and more. He also spoke about WSO2’s vision for the connected enterprise and its future in the middleware industry.

Sanjiva

Other keynotes for the day included an engaging presentation on Vega, the high performance electric sports car being developed in Sri Lanka. Dr. Harsha Subasinghe, the president and CEO of Codegen, and Dr. Beshan Kulapala, a research scientist at Codegen highlighted the challenges and opportunities in leading complex engineering projects to success.

Codegen

Dr. Frank Leymann, a director in the Institute of Architecture of Application Systems at University of Stuttgart, discussed loose coupling and its implications on microservice and cloud native architectures.

Frank Leymann

Isabelle Mauny, the vice president of product management at WSO2, gave the final keynote for the day on how analytics can improve customer experiences. Interestingly the phrase “Turkeys” was trending on our twitter dashboard during her talk!

The day continued with dedicated tracks running in parallel on cloud, API management, security and integration.

Several customers shared their experiences and how they have used the WSO2 Platform to effectively meet their technical and business goals.

Harshavardhan Mohanraj and Praveen Doddamani, technical leads at ZeOmega, spoke about how they leveraged the WSO2 platform to build a healthcare solution while improving component manageability and standardizing security in the API Management Track.

Zeomega

Ibrahim Khalil, a system integration analyst and team lead at Capgemini, spoke about how they leveraged their experience with United Nations agencies and built a vertical solution, enabled by WSO2 products, for UN organizations in the Cloud Track.

Ibrahim Khalili

Gina Keune, the team lead of integration and configuration at Royal Automobile Association, shared the challenges they faced and wins they celebrated when incrementally adopting SOA using WSO2.

Gina Keune

Charith De Silva, a lead architect at WSO2.Telco, introduced WSO2.Telco IDS which provides a fully Mobile Connect (OIDC) compliant solution for telcos embarking on a federated ID strategy.

Charith De Silva

This year we also hosted a special Strategy Forum for CxOs which saw attendees from companies such as Honeywell, NYU, John Keells Holdings, zMessenger, LOLC and more. The forum was led by WSO2 VP of Solutions Architecture Asanka Abeysinghe who spoke about the digital transformation of enterprise platforms.

Strategy Forum

The day ended with a networking event featuring smooth jazz tunes provided by Brown Sugar. Tasteful bites accompanied by cocktails coupled with lounge-like seating made it the perfect environment to catch up with industry experts and peers.

Networking event

The third and final day of the conference was also packed with insightful technical sessions. The tracks covered topics on governance, IT consumerization, analytics, devOps and app development.

The session on microservices attracted over a 100 of the attendees and proved to be one of the most popular talks of the day.

Azeez

Another popular talk was by Kiran Kumar, an enterprise architect at Wipro, who discussed a case study of a governance system where service governance meets API governance.

Kiran Kumar

The technical sessions came to an end with a closing keynote by Asanka and panel discussion on the benefits and effects of creating a digital enterprise.

Asanka Panel

In addition to the technical sessions, a team led by Srinath Perera, vice president of research at WSO2, showcased capabilities of the WSO2 Analytics platform by hooking up with Twitter to create a sentiment analyser. The project which tracked #wso2conasia was able to give valuable insights into sessions and popular topics. It also helped identify the most dedicated tweep, who walked away with a GoPro camera for his contributions.

Sentiment analysis

Sumedha Rubasinghe, director of API architecture at WSO2 also ran a project that combined  Google’s voice API with the soon to be released WSO2 IoT Server which displayed what speakers of each session were talking about.  This was just a sampling of what you can expect in the future with WSO2’s IoT Platform.

Voice Analyser

Source: Readme.LK

In true WSO2 style, WSO2Con Asia 2016 came to a close with a rocking after-party featuring the band Glory.

Party

In the coming weeks we’ll be sharing more details of the presentations, so stay tuned for that. In the meantime you can check out the slides for all sessions at the WSO2Con Asia website.