Tag Archives: ESB

Meet the WSO2 Enterprise Service Bus 5.0

The WSO2 Enterprise Service Bus is renowned as a lightweight, 100% open source ESB; it helps everything from big, digital-first companies (like eBay) to transnational governing bodies (like the Eurasian Union) achieve the kind of enterprise integration scenarios they need to work. We’ve now released version 5.0 of the WSO2 ESB – and here’s why you should take a look. 

As part of our new product strategy, the WSO2 ESB 5.0 release is composed of three key components – the runtime, tooling, and analytics. We’ve also added several new, major features:

  • Mediation Debugger for error detection in mediation flows
  • Data mapper to convert and transform data
  • Mediation statistics and tracing using ESB Analytics component
  • Support for JMS 2.0 and WebSocket transports

Interested? Let’s explore in more detail.

WSO2 ESB Tooling 5.0

The new WSO2 Developer Studio-based ESB tooling plug-in is built to work on top of Eclipse Mars2, the widely used IDE. It offers developers with improved usability and brings some new capabilities to the table:

Mediation debugging

At the operational level, a series of mediators called sequences determine the behavior of messages. Users were previously unable to figure out if these mediators -including their inputs, processing, and outputs – were operating as intended. The newly introduced Mediation Debugger allows users to debug this message mediation flow proactively, to identify any errors and rectify them in the development environment itself (figure 1).

image05Figure 1: Debugging mediation flow

Data conversion and transformation

 

WSO2 Data Mapper is one of the most awaited features by our customers. Having been built as a mediator that can be included in the mediation flow, it is capable of converting the data format of a message between JSON, XML or CSV (figure 2). It also comes with an operations palette that supports a number of operations to help users perform different functions in data mapping (figure 3). For instance, arithmetic functions enable users to add, subtract, multiply, divide or carry out other functions on data as a part of its transformation. Users can easily drag and drop operators into the mapping diagram and edit on the go.

The main categories of operations are listed below.

  • Common
  • Arithmetic
  • Conditional
  • Boolean
  • Type conversion
  • String

We’ve made it more appealing to developers by introducing a graphical mapping experience that also generates the execution files required to run on the Data Mapper Engine.

image09Figure 2: Data mapping

What’s special about WSO2 Data Mapper is that it can function independently of other WSO2 products or components.

image00Figure 3: Adding operations in data transformation

WSO2 ESB Analytics 5.0

The analytics component is a new addition to the ESB that lets users publish information on ESB processes. Previously, users had the ability to connect WSO2 DAS and define their own ESB dashboards. However, we’ve made the experience better by providing an analytics dashboard out of the box, with pre-defined statistics for ESB artifacts such as proxy services, APIs, sequences, endpoints and inbound endpoints. Users have the option here to choose which artifacts’ statistics and/or tracing should be published by enabling or disabling this through the runtime instance.

The dashboard displays a summarized view of the requests processed, their success/ failure rates, throughput, and message count over a specified time period, and top artifacts by request count (Figures 4, 4a, 4b, 4c respectively). For example, users can identify the top sequences, endpoints or the top APIs by request count in separate gadgets.

image04Figure 4: Statistics dashboard

image03Figure 4a: Overall transactions per second graph

image01

Figure 4b: Message count graph

image08

Figure 4c: Top endpoints by request count graph

The dashboard provides many features – from configurable timelines and detailed stats by artifact type to visual additions, such as theming and the customizability to portray your brand and explain statistics (figure 5).

image07

Figure 5: Dashboard with configurable timelines, theming buttons and navigation pane

Tracing mediation flows

Not only is it important to monitor statistics, but the ability to drill down into anomalies encountered in ESB processes is essential for DevOps teams, especially in production environments. Hence, we’ve introduced message tracing capabilities to provide more visibility into message mediation flows so developers can identify problems in live environments.

For a particular artifact type such as an API or proxy service, operational teams can dive into obtaining overall message count, latency and processing details of each message. Consider a scenario where messages pass through a REST API: the message gadget (figure 6) helps DevOps teams proactively identify which messages failed, as shown below.

image06Figure 6: Messages and their status

We’ve also made it possible to further scrutinize into the message flow (figure 7), and dive into a mediator and view its properties in relation to a payload, transport properties and context properties (figure 8), both before and after the mediator. It provides a mechanism by which operational teams can verify if message flows operate as they are intended, and fix any errors.

image02Figure 7: Message flow detailed view

image10Figure 8: Mediator properties

These features also are useful in testing environments to evaluate changes done to artifacts (sequences, APIs, endpoints etc ) or message mediation flows, as a feedback mechanism prior to moving into production.

Support for JMS 2.0 and Websocket

Last but not least, we’ve also added two transports – JMS 2.0 and WebSocket – to be able to support more integration use cases.

With JMS 2.0, the WSO2 ESB supports new messaging features – shared topic subscription, getting the JMSX delivery count and specifying message delivery delays. A typical scenario would be using the JMS endpoint to act as a shared topic listener so that it can connect to shared topic subscribers and help share the workload. We’ve also introduced WebSocket as a transport and inbound protocol to support bidirectional message mediation mostly advantageous in near-real-time communication for IoT, web, and mobile scenarios.

We’ve bumped the product quite a bit in this release and we’d love to hear what you think about the all new WSO2 ESB 5.0.

Go ahead and try it yourself – it’s free to download and use!

R.I.P., Gateway Server; Hello, Gateway Framework

Last November, we announced a high performance, lightweight, and configuration-driven message gateway – WSO2 Gateway. This product provided fully decoupled protocol handling layers and message processing layers, making it easier for users to configure messaging behaviour at each layer independently. We made it available as an Alpha version, with a plan to announce general availability this year.

A few months later, and as we progressed with our GA release plan, we realized there was a broader need: to instead use this component as a framework for other WSO2 products which use such a gateway pattern. The Gateway Framework will therefore become the core of other gateways, such as the API gateway of our API management offering, and power the next generation of our Enterprise Service Bus.

The Gateway as a Framework

The Gateway pattern is commonly applicable in various products in WSO2 platform – including the ESB, API Gateway, Data Integration Server and Identity Bus. Thus, we introduce the Gateway Framework, so that we can build similar products on top of it; for instance, the next generation Integration Server will be built on top of this Gateway framework.

image01

The key points are:

  • The Gateway is no longer a product but a core framework that provides generic message mediation capabilities.
  • Carbon Transport and Carbon Messaging provides the decoupling between transport and message mediation/processing layers, serializing and deserializing wire-protocol message to/from generic message representation.
  • The Gateway Core is a message-mediation-engine implementation that can receive/send Carbon-Messages from/to Carbon Transport and mediate messages. It provides the common runtime to all the products that share the Gateway characteristics.

Why WSO2 Gateway?

We built WSO2 Gateway with several goals in mind.

  • Pluggable mediation engines : With this new architecture, we were able to plug-in any message mediation engine and were able to use the transport components to do generic protocol handling tasks across the entire platform.
  • Fully decoupled protocol handling layers : Decoupling protocol handling/transport component from mediation engine is the key to building an integration solution that can deal with numerous protocols and message formats.
  • Super fast messaging capability : We built a fast and low latency HTTP transport component based on Netty, LMAX Disruptor and WSO2 PassThrough messaging architecture.

As the initial mediation engine implementation we used Apache Camel alongside the Netty based WSO2 HTTP transport. We’ll be moving away from Camel and Synapse now.

Moving away from Camel and Synapse

Although we initially used Apache Camel as the message mediation engine, we realized that neither Apache Camel nor Apache Synapse would be the ideal fit for our next generation integration products.

  • Apache Camel doesn’t give us a significant advantage as a mediation engine. It was intended and built with a centralized, monolithic and heavy-weight ESB in mind.
  • We need to build a lightweight and container friendly mediation engine runtime. Neither Synapse nor Camel will suit our need needs for lean (memory footprint, CPU consumption and startup time), high performance, low latency integration solutions.
  • The Gateway will be the foundation of our next generation of integration solutions for the next decade. If we move along with Camel or Synapse, we are literally using technologies that were built almost a decade back, and we won’t be able to leverage cutting edge technologies such as the power of Java 8 and new reactive programming paradigms.

Building a new mediation engine from scratch gives us the opportunity to leverage cutting edge technologies to build a better solution.

New mediation engine and visual model  

One of the key objectives of the new mediation engine is to envision a better visual modeling and visual representation of message mediation logic. We’re trying to move away from conventional flow diagram-like models (that most existing products are based on)  and inventing a much simpler, yet a much powerful way to represent complex message mediation logic.

  • The new visual representation that we are using is inspired from sequence diagrams, and we’re using the textual representation of the same visual model as the mediation language.
  • Flow designing, data mapping and debugging will be an integral part of this visual tool.

The final question: Migration

How do we migrate from existing ESB to the next generation Integration Server?

As this is a complete redesign and revamp, the migration from existing ESB to the new integration server won’t be a straightforward task. Although we can’t really guarantee a 100% seamless migration, we’re planning to build automated migration tools to migrate the existing configuration to the new one with as much accuracy as possible.

Transform Your Enterprise IT: Integrate and Automate

Most enterprises deal with a variety of common IT problems to which they would find quick fixes. One such example is the need to maintain five different usernames and passwords to login to five different systems. Another typical example is the closing of a sales deal – the sales department would conclude the deal and ensure the goods are delivered; this would be updated on the sales records, however, when the finance department reconciles invoices against sales at the end of the quarter, there might be mismatches because the invoicing process was missed.

area-review

To address these issues, most enterprises will use a combination of basic IT and collaboration software to manage day-to-day requirements. And over time, these requirements will change, prompting a slight shift in the enterprise’s IT landscape too. This may result in a situation where different teams within the organization will find the most efficient ways to carry out tasks and meet their IT requirements with the use of packaged software, possibly by building their own, or even subscribing to more SaaS-type offerings.

While this might temporarily fix specific problems, it will pose long-term challenges as such measures are often not pre-planned or do not follow a particular IT roadmap. The actual negative effects of individual teams working in silos would only be felt when the company starts to grow and the use of various systems increase as well. Eventually, the use of several systems that don’t talk to each other will cause operational issues and even hurt motivation among employees.

The recurrent problems with these multiple systems working in silos include extensive manual effort, errors, blame, rework, frustration, complaints, and the need to manage multiple passwords. These in turn result in inefficiencies.

To address these challenges, the enterprise needs an easy-to-implement, cost-effective solution. There’s no guarantee though that there would be a plug and play type of system or one that could be customized to meet the enterprise’s exact requirements. The enterprise would seek a unique, bespoke solution that would either mean they change the way they work with existing software or rethink the software itself.

The most viable option would be to integrate the systems (which, of course, have proven to be efficient to meet a specific requirement) used by different functions and then explore some sort of automation that will provide relief to employees.

WSO2’s highly-acclaimed open-source middleware platform has the capabilities that enable seamless integration of IT applications, thus streamlining day-to-day business activities of a given enterprise. This in turn will boost efficiency and integration across business functions and teams and improve overall productivity as well.

For instance, WSO2 Identity Server (WSO2 IS) can define an identification for a user in a particular organization, enabling him/her to log into multiple systems on-cloud or on-premise with a single username/password.

The enterprise too will benefit as WSO2 IS offers provisioning capabilities that allow your IT to register and auto-provision new employees across multiple systems as well as easily de-provision them when they leave the organization.

WSO2 Enterprise Service Bus can meet all your integration challenges with its capability to connect various systems that speak different languages. It also comes with a defined set of connectors to further support integration of systems, be it on the cloud or on-premise.

Once all of your systems have been integrated, you can leverage WSO2 Data Analytics Server (WSO2 DAS) to pull reports from different functions within your organization and automatically collate data that will translate to valuable information required to make business decisions. WSO2 DAS has in-built dashboard capabilities that will automatically create and publish dashboards on a real-time basis.

Moreover, all WSO2’s products are 100% open source, which gives enterprises the freedom of choice and empowers the business with limitless possibilities to expand.

Learn more about WSO2’s comprehensive and open platform for your connected enterprise.

For more details on how to establish friendly enterprise IT and get more love from your team, watch this talk by WSO2’s VP Operations, Shevan Goonetilleke.

Orchestration and Choreography – When To Use An ESB vs a Workflow Engine

Service chaining and workflows, commonly referred to as orchestrations, are common integration scenarios in enterprise systems development. There are two distinct type of orchestrations we deal with when realizing these systems:

  1. Short-running stateless orchestrations
    Service chaining messaging patterns that are more synchronous in nature and deals with transient data/sessions.
  2. Long-running stateful orchestrations
    Service chaining messaging patterns that are asynchronous in nature. These workflows take human input and approvals and are configured to run for longer durations with more persistent sessions.

If we look at these two types a bit closely the short-running ones are more straightforward; it mainly follows a request/response pattern, talks to multiple service endpoints, derives a response from one service call and composes the message to be sent to another service call. Enterprise integration patterns, such as message splitting, transformation, cloning and aggregation, are key building blocks in such orchestrations. The enterprise service bus (ESB) pattern fits this description quite well.

Screen Shot 2016-03-29 at 12

However, the long-running ones are more complex in nature and often involve human approval activities, work delegation, and assignments. A typical example for such a use case is a loan approval workflow; when a banking customer applies for a loan, the request triggers multiple services (1) credit approval service – this means connecting to multiple credit bureaus and valuing the customer’s credit worthiness, (2) evaluating the loan type (personal/home/vehicle, etc.), (3) finally, human approval (bank manager’s or loan department’s approval).

If you look closely, this particular workflow has different types of system interactions from a technical point of view. Not all interactions are long-running, yet the end-to-end scenario is so. The traditional architectural belief for such an end-to-end scenario is to model it with BPEL or BPMN (with a workflow/BPM tool), resulting in a complex model with human tasks and external links.
The common characteristic of this use case is that it is asynchronous in nature. The loan applicant will not get a response instantaneously; other parts, such as calling a credit bureau and getting the credit rating, is a synchronous stateless service call. Manager approval is a human interaction. Hence you do not have to follow the traditional path and build the entire workflow with the BPM tool. What you can follow is a hybrid model where you model the synchronous short-running stateless interactions with an ESB and long-running human interaction with a BPM tool. The proposed solution is depicted in the following architecture diagram.

Screen Shot 2016-03-29 at 12

The corresponding sequence of interactions has been illustrated in the following diagram.

Screen Shot 2016-03-29 at 12

After Sonic ESB pioneered, WSO2 ESB continuously innovated

After Sonic ESB pioneered the Enterprise Service Bus market and created technology to more readily integrate applications using Enterprise Integration Patterns, WSO2 continually added innovations to the core ESB pattern and created the highest performance, lowest footprint, fully interoperable and flexible ESB, WSO2 ESB.

Progress Software recently lowered their sights on supporting enterprise integration developers, and Progress is has publicly announced intent to sell Progress Sonic ESB, Actional services management, and FuseSource. Progress’ recent strategy shift places Sonic ESB, Sonic MQ, Actional, and FuseSource customers at risk of further obsolescence.

Where did Progress’ corporate strategy fail? By growing through ad-hoc acquisition and superficial integration instead of organic platform development. Progress played a failing game espoused by IBM, Oracle, and Software AG. Progress acquired competitive threats and then invested the minimal amount required to create a ‘SOA Suite’. As often the case, the strategy led to multiple overlapping products, competing business units, lost internal talent,
and a disjointed customer experience.

At WSO2, we have consistently taken a different approach. Our complete, composable, and cohesive platform was built through organic development, which continually enhances our core platform, incorporates market-leading innovations, and delivers a seamless customer experience. Our WSO2 Enterprise Platform enables complete data to screen enterprise integration, SOA, BPM, API management, web application development, and Cloud.

Our WSO2 ESB, WSO2 Governance Registry, WSO2 Business Activity Monitor, and WSO2 Identity Server provide a production proven, high performance SOA middleware foundation. We welcome you review our case studies and learn how WSO2 ESB processes more than 1 billion transactions per day for eBay, streamlines the development and maintenance of smart power grids, supports T24 core banking systems, and enables consolidated reporting across
enterprise applications.

Our Offer to Progress Sonic ESB and Progress FuseSource Customers

WSO2 desires to assist Progress Sonic ESB and Progress FuseSource customers choose a viable, stable, and supported middleware platform. We are offering free Evaluation Support to current Progress Sonic ESB and Progress FuseSource customers, and would be pleased to demonstrate how our market leading WSO2 Enterprise Service Bus and WSO2 SOA Platform meets your evaluation criteria. Feel free to contact us via our Progress customer offer landing page, our contact form or send us an email note.

Chris Haddad, VP Technology Evangelism Chris’s blog: http://blog.cobia.net/cobiacomm/