The Evolution of Integration: A Comprehensive Platform for a Connected Business

Today, successful enterprises rely heavily on underlying software applications. To meet prevalent diverse business requirements, enterprises have to pick and choose different software applications that have been created with disparate technologies and built by different vendors. When the enterprise needs to fulfil a given set of business requirements, most often they have to make these disparate software applications work together to produce a unified set of functionality. The task of plumbing different software applications and forming new software solutions is referred to as 'enterprise integration'.

Enterprises often encounter many challenges when dealing with applications built by varied vendors; one of the most difficult, yet common problems is that some of these applications are based on proprietary standards and others built with open standards. These challenges can be addressed with the use of architectural styles and enterprise integration technologies that lie within a comprehensive integration platform.

There are several integration styles that can be used to integrate two or more software applications. In the publication "Enterprise Integration Patterns" by Gregor Hohpe, these integration styles have been illustrated under four main categories: File Transfer, Shared Databases, Remote Procedure Invocations, and Messaging (Figure 1).

Managed File Transfer has been around ever since the early stages of enterprise integration where one application writes a file that the other application reads and visa versa. The file format, standards, location, privileges and read/write coordination must be negotiated between the two applications beforehand [comma-separated values (CSV), electronic data interchange (EDI), electronic protected health information (ePHI), etc. are still used in enterprise IT].

With the Shared Database approach, software applications can use a common database to share data among each other.

In Remote Procedure Invocation (remote procedure calls, or RPC), a given application can expose some functionality via a well-defined interface and other applications can invoke such functionality via a remote invocation. Technologies, such as common object request broker architecture (CORBA), were popular as an RPC-style integration methodology prior to SOA and web service technologies.

With Message-based integration style, all applications connect to a unified messaging bus and exchange data and invoke functionalities via messages. It allows complete decoupling of applications and enables high speed, asynchronous/synchronous, program-to-program communication with reliable delivery message-oriented middleware (MoM), service-oriented architecture based on web services and an enterprise service bus.

While these integration styles are actively used in most integration scenarios, the enterprise integration space has evolved around these styles.

Over several decades, enterprise integration technologies have leveraged one or more of the integration styles that were discussed earlier. The evolution of enterprise integration middleware was driven by the evolution of enterprise requirements.

3.1 Homegrown Integration Solutions/Custom Development

In the early stages of enterprise integration, most organizations met their integration requirements by implementing in-house integration middleware. With this approach, integration middleware was developed in-house using ad hoc and proprietary protocols.

Over time, with the addition of multiple software systems, the home-grown middleware system becomes increasingly complex and ultimately fails. Most often than not, these systems are monolithic, not reusable, and built in an ad hoc manner with point-to-point connectivity between each application. With this approach, organizations end up focusing more on solving its integration problems rather than focusing on the business functionality of the solution.

3.2 Enterprise Application Integration (EAI)

It is near impossible to build and maintain a homegrown integration solution for complex integration scenarios that involve several applications and systems. Therefore, there was a need for a dedicated integration middleware framework that can centralize and facilitate all integration requirements of the infrastructure.

To this end, an EAI solution helps to seamlessly integrate applications and achieve a given business objective. Initially, EAI solutions were designed based on hub-spoke architecture and later with bus architecture.

3.2.1 Hub/Spoke Architecture

EAI hub/spoke architecture simplifies the integration of various applications by avoiding point-to-point interactions and allows the loose coupling of applications. Since the hub is the centralized component, this reduces the overhead of maintaining multiple configuration repositories. As illustrated in Figure 2, the central broker (hub) connects all applications using adapters (spoke) for each software system.

Yet, the hub-spoke architecture has inherent limitations; the centralized hub introduces a single point of failure, and with the heavy load, the hub can be the main bottleneck of the entire infrastructure. Another drawback of this architecture is that most of the solution was built with proprietary technologies and focussed on a particular business domain. Hence, it was difficult to adopt EAI as a common integration middleware.

3.2.2 Bus Architecture

On the other hand, ‘bus architecture’ as illustrated in Figure 3 looks to address some of the limitations prevalent with EAI hub-spoke.

Bus architecture solves the scalability issues of the hub-spoke architecture as the centralized messaging bus can be scaled horizontally. However, the distributed integration tasks complicates and increases maintenance and troubleshooting overhead. Again, the non-standard base and the proprietary nature of the EAI bus architecture eventually hampered the adoption of EAI-bus.

3.2.3 SOA and the Advent of Enterprise Service Bus

From the 2000s, SOA has become increasingly popular as an architectural strategy and many organizations adopted SOA as the foundation of their enterprise architecture. SOA was realized in the form of web services and web services became a primary part of implementing software solutions.

This has revolutionized the way enterprises approach their software solutions today. The heavy adaptation of SOA leads the industry to seek a solid infrastructure to build a robust SOA. In this context, software applications are more or less replaced with web services and the necessary infrastructure is required to do the plumbing among these services to form various business solutions.

As with software applications, the interactions between services can be implemented with pointto-point architecture, which will have the same set of problems and limitations that were discussed earlier. With SOA, the bus architecture of the EAI has further evolved to a new architectural concept, the ‘enterprise service bus’ (ESB). The idea of using an ESB is to provide a unified, standard-based messaging layer where all the interactions between services take place.

ESB is designed as the middleware layer that enables interoperability among heterogeneous systems and services using the SOA model. ESBs are often fully compliant with open standards such as WS-* standards and encourage the organization to implement business functionalities as services. It acts as the main messaging backbone in any SOA implementation. Furthermore, unlike traditional EAI, the messaging layer is lightweight and can be horizontally scaled as per the organization’s needs.

As explained in Figure 4, an ESB eliminates point-to-point integration of the organization and integrates all disparate systems with the bus architecture. Generic ESB functionalities can be listed as follows:

• Message mediation - Manipulates the message content, direction, destination, and protocols with message flow configurations
• Service virtualization – Wraps existing systems or services with new service interfaces
• Protocol conversion – Bridges different protocols, e.g. JMS to HTTP
• Support for enterprise integration patterns (EIP) – EIP is the defector standard for enterprise integration (https://www.eaipatterns.com/)
• Quality of service – Applying security, throttling, and caching
• Connecting to legacy and proprietary systems – Business adapter including SAP, FIX, HL7
• Connectors to cloud services and APIs – Salesforce, Twitter, PayPal, and more
• Configuration driven – Most functionalities are driven by configuration, but not code
• Extensibility – There are extension points that can be used to integrate with any custom protocol or a proprietary system

With the advent of ESBs, most EAI vendors simply rebrand their EAI solutions to an ESB solution. Yet, there are a few ESB implementations that have been built from scratch as lightweight, standard-based, and interoperable ESBs.

4.1 Enterprise Application Integration (EAI)

The use of SOA as an architectural style and ESB as the infrastructure for implementation has been a great success for most organizations. In recent times, most organizations have solved their integration problems by leveraging ESB as the backbone of their IT systems. In addition, the way an organization implements IT solutions was revolutionized and business functionalities are implemented as services rather than applications. Hence, in general, using an SOA/ESB for enterprise integration needs is a good move.

However, the enterprise IT space has seen some drastic changes; the four disruptive forces of enterprise IT (i.e. mobile, data, social networking, and cloud) have prompted organizations to rethink and redesign enterprise IT solutions to cater to modern enterprise needs. Just the SOA/ESB approach cannot facilitate all of these new enterprise IT requirements.

SOA and ESB are primarily designed for internal interactions. Given the innate complexity of SOA/ Web services and ESB, the cost of cross boundary business couldn’t be significantly reduced. Owing to limitations, such as complex service contracts, non-mobile friendly data formats, inability to carry out frequent iterations and not able to support service versioning, accessibility limitations, and the absence of monitoring and analyzing, the modern enterprise needed enterprise IT solutions that looked beyond SOA and ESB.

4.2 Exposing Business Functionalities via Managed APIs

The disruptive forces of mobile, data, cloud, and social networking have propelled the modern enterprise to a new level. Most enterprises found that their business functionalities are accessed mainly via mobile apps than the regular web portal. The proliferation of apps motivates enterprises to expose business functionalities in a more convenient way than services to enable mobile access. Hence, enterprises need a simple, secured, and managed approach of exposing business functionalities.

As a result of sheer demand, API management has emerged as a way of exposing business functionalities in a managed, accessible, monitored, and adaptive way.

APIs in general are the business functionalities offered over the network on top of standard protocols (mostly HTTP), with well-defined, but loose contract and with simple message formats.

However, it doesn't solve all the SOA/ESB limitations. Still, you can leverage APIs to overcome SOA/ESB limitations by converting APIs to ‘managed APIs'.

Managed APIs are publicly advertised and enables users to subscribe and access the API (self-service). They also support versioning and service-level agreements and can be secured and authorized to ensure protection. Moreover, managed APIs can be monitored and monetized, thereby offering further benefits to the enterprise in terms of enhancing business processes and increasing revenue.

4.3 SOA, Integration, and API Management: The Close Cousins

Enterprises can leverage the power of its integration platform and the power of API management by combining SOA and API. In this regard, there are some key elements that should be noted.

• APIs cannot replace integration: There is a myth that APIs can replace integration, but APIs primarily expose business functionality in a managed fashion; in other words, it does not address any problems when dealing with disparate systems and technologies.
• Integration of internal services, systems, data and cloud APIs: To expose a business functionality as an API, any organization would need to have an underlying implementation of a service and the plumbing between all the required systems and services. For this, you would need an integration backbone, such as an ESB, and the API management layer would sit on top of it.
• Unable to mangle SOA for API management needs: Some vendors try to mangle SOA frameworks to support API management. SOA is not designed to meet the requirements of API management and often ends up with disconnected and unstable frameworks.

So, how do you build real-world enterprise IT solutions by leveraging ESB and API management? That’s where the API façade pattern comes into play.

API Façade Pattern

The use of SOA, integration, and API management in combination to realize business requirements is a very common technical requirement for most organizations. API façade is the standard pattern that addresses those requirements. It's a simple, but extremely useful pattern as it exposes a business functionality without the underlying technical complexities.

The integration landscape is looming over a storm that’s driven by the new forces of enterprise IT; cloud, mobile, big data and social. Enterprises are moving beyond the traditional means of integration and there is a proliferation of enabling services through social and mobile platforms and offloading some part of the on-premise business to the cloud through software-as-a-service (SaaS) solutions.

Therefore, organizations need an integration infrastructure that can support current and imminent integration needs. Hybrid integration platforms are a new way to connect cloud based, mobile and on -premise systems. It involves a mix of on-premise, cloud, B2B, social, and mobile integration scenarios of varying complexity, and only a suitable combination of on-premise and cloud-based integration platforms can fulfill such complex requirements. A hybrid integration platform is a federation of two major technologies.

1. SOA/ESB based, on-premise integration platform: This is a base set of features that include communications, mediation, orchestration, transformation, QoS, security, monitoring, administration, and management.
2. . Integration platform as a service (iPaaS): iPaaS is a cloud hosted integration platform designed to consume cloud resources exposed as services. iPaaS is realized as a suite of cloud services aimed at addressing a wide range of cloud, B2B, and on-premise integration and governance scenarios.

As future demands become increasingly oriented toward iPaaS-based solutions, based on the complexity and the type of integration scenario, iPaaS has been developed into different categories of integration. There are a few major areas of iPaaS that can be identified in future hybrid integration middleware solutions. Thus, organizations need a comprehensive integration platform to meet today’s modern enterprise integration requirements. The WSO2 integration platform has helped many enterprises meet their integration requirements.

In the modern enterprise IT world, most organizations want to convert their business to one that’s connected. A ‘connected business’ allows customers, partners, employees, internal, and external systems, as well as any other parties to deeply engage and smoothly collaborate with the enterprise. The cost of transactions between the enterprises also reduces significantly, while the enterprise, its customers, and partners gain a competitive advantage over conventional enterprises. The WSO2 middleware platform enables companies to build a connected business by providing comprehensive, data to screen open source products that span the entire breadth of SOA, yet remain lean, simple to use, and are cost effective.

6.1 Building an Integrated Business with the WSO2 Platform

The foundation of building a connected business is to have a solid integration backbone to build an internally and externally integrated business. The WSO2 integration platform has the capabilities for an enterprise to achieve this; the platform has a logical categorization of WSO2 middleware products that are used in the enterprise integration space, as illustrated in Figure 5.

WSO2 Enterprise Service Bus (WSO2 ESB) is the main integration backbone of the WSO2 platform, which can be used to integrate anything with everything ranging from web services and RESTful services to complex SAP-based integration scenarios.

WSO2 Data Services Server (WSO2 DSS) is primarily used to dynamically wrap existing heterogeneous and disparate data stores with a service layer. WSO2 DSS can integrate data sources (RDBMS or NoSQL), create composite data views, and host data services. With WSO2 DSS, you can completely decouple your data stores from internal and external consumers of those data stores.

WSO2 Business Process Server (WSO2 BPS) enables developers to easily deploy business processes written using the WS-BPEL standard and also serves as the business process management and hosting environment. In addition, it supports human tasks where the business processes that need human interaction are required. In contrast to WSO2 ESB, WSO2 BPS is designed for stateful and long-running business processes.

WSO2 Message Broker (WSO2 MB) enables applications to exchange communications asynchronously or publish messages for timely access by many subscribers. WSO2 MB supports JMS and AMQP standards for persistence messaging and can be used in scenarios where guaranteed message delivery is vital.

In addition to the above-mentioned products, WSO2 Application Server (used to host web services) and WSO2 Governance Registry(registry and repository and SOA governance) are also used as supporting products in this space. delivery is vital.

6.2 Using API Management on Top of the Integration Layer

Once the functionalities are decomposed into services and they are externally and internally integrated, you need to expose those functionalities as managed APIs to your customers, partners, and internal parties, among others. For this, you can use the API façade pattern to design the solution and WSO2 API Manager can be incorporated as shown in Figure 6.

The API management layer comprises the API publisher where the organization can publish its APIs; the API store to expose the APIs to internal and external application developers; and the API gateway, which is capable of serving the the actual API traffic from the API consumers. WSO2 Identity Server is used to support authentication and authorization, and WSO2 Business Activity Monitor or WSO2 Complex Event Processor are used for monitoring and analytics.

6.3 Hybrid Integration with WSO2

WSO2 offers its iPaaS solutions in two main streams. The ESB capabilities that we discussed in the previous sections are offered in the form of ESB as a Service offering in WSO2 Integration Cloud. In addition to ESB as a service, the pre-built integration scenarios are available as ‘Integration Templates.’ WSO2 iPaaS offerings use ESB behind the scene in all integration scenarios and extensively leverage WSO2 ESB 'Connectors' (https://storepreview.wso2.com/store/) to intetegrae with cloud APIs.

In keeping pace with the evolution of enterprise integration, enterprises have to rely on innovative solutions to be competitive. Mobile, data, cloud, and social networking too have created a greater need for modern enterprises to seek cutting-edge and comprehensive solutions to address their enterprise integration requirements. Organizations today require integration infrastructure that can seamlessly connect cloud-based, mobile, and on-premise systems. The key to achieving these is to adopt a solid integration platform to build an internally and externally integrated business that’s future proof and able to adapt to any new requirement. The WSO2 integration platform’s logical categorization of WSO2 middleware products enables enterprises to meet these needs in a costeffective and timely manner. WSO2's single code-based platform also enables enterprises to leverage the advantages of combining SOA and API management.