Table of Contents
Integration teams often adopt different architectural styles for integration, either using a centralized ESB or a more distributed cloud-native, microservices-based approach. Current integration tools and approaches often force teams toward one approach or the other — making it difficult for IT groups to agree on a single technology, vendor, or organization. Moreover, conventional integration technologies are neither agile nor developer friendly. On the other hand, general-purpose programming languages and frameworks are developer friendly but overwhelmingly complex to be used as an integration technology. Such divisions between development and integration approaches slows time to market for an increasing number of IT projects reliant on integration. This is the heart of the “integration gap”.
To address this integration gap, WSO2 has come up with Enterprise Integrator 7.0 (EI 7.0), which is an open-source, cloud-native, and hybrid integration platform for APIs, data, and event streams. Graphical configuration-driven integration and code-driven integration are two approaches for creating integrations, and EI 7.0 is the only solution to provide support for both, plus a choice of styles for decentralized cloud-native microservices or centralized ESB style architectures. Furthermore, the solution is a core component of the WSO2 Integration Agile Platform, which also includes API management and identity and access management capabilities.
1. The Integration Problem and the Enterprise Service Bus
Even if you are a company that starts today, an integration problem will arise. For example, a common need is to integrate the customer relationship management (CRM) system with the legacy enterprise resource planning (ERP) or accounting system in order to link financial information to assist customer services. There will always be a need to connect to another service. The problem will never go away. Initially, a small business will use software applications for various needs, and most of the time, these applications will reside in silos. As businesses grow, they can no longer afford to have them in silos. When stakeholders need to understand what’s going on in the business, they need to integrate all the apps that drive their digital businesses to combine data and services from within the organization, throughout their external ecosystems, and across a range of devices. This is why application integration is fundamental.
Integrating APIs, services, data, and systems has long been one of the most challenging yet essential requirements in the context of enterprise software application development. We used to integrate all of these disparate applications in a point-to-point style, which was later replaced by enterprise service buses (ESBs) alongside service-oriented architecture (SOA). The ESB was used as the central bus to connect and integrate all the disparate APIs, services, data, and systems. If a given business use case required talking to different entities in the organization, it was the job of the ESB to plumb all these entities and create composite functionality. Hence, ESB solutions were usually powerhouses of all the built-in integration capabilities, such as connectors to disparate systems and APIs, message routing, transformation, resilient communication, persistence, transactions, and various other enterprise integration patterns (EIP)s.
Figure 1: A typical implementation of the centralized monolithic ESB pattern with WSO2
But, with the cloud-native microservices architecture (MSA) coming into the picture, the centralized ESB was considered an anti-pattern because any system that requires a lot of central management can become problematic as with any monolith. With the ESB replaced with smart endpoints and dumb pipes, the core microservices had to take care of all the application integration. However, a new set of challenges for enterprise application integration was introduced with this approach. The obvious tradeoff from the complete decentralization of the smart ESB is that the code complexity of the microservices dramatically increased, as they had to cater to these application integrations in addition to the service’s business logic. These integration requirements ranged from the integration of several microservices to creating anti-corruption layers. Furthermore, even with the adoption of microservices, there are applications that are still suited for a monolithic architecture; therefore, brownfield environments continue to exist. So, in most greenfield and brownfield scenarios and with evolving business needs, the capabilities of an enterprise service bus in an SOA are still required in an MSA to create a cohesive experience.
2. The Shift from the Monolithic ESB to Integration Microservices
As an alternative to a centralized ESB, implementing integration microservices is the best option from an MSA point of view. In this approach, integration or composite services will be hosted as a second layer of microservices. These services often have to support the required ESB functionality such as routing, transformations, orchestration, resilience, and stability patterns. As opposed to the monolithic ESB approach, all ESB/integration functionalities are segregated and dispersed among all the integration microservices.
Core/atomic microservices are placed at the bottom layer. They are extremely fine-grained, self-contained services (with no external service dependencies) comprising the business logic with little or no network communication logic. The integration microservices are more coarse-grained than atomic services and independent from each other. They contain routing logic, data mapping logic, and network communication logic, which handles inter-service communication through various protocols and resiliency behaviors such as circuit breakers. These services also could bridge the other legacy and proprietary systems (e.g., SAP ERP), external web APIs (e.g., Salesforce) and shared databases (often known as the anti-corruption layer).
Figure 2: Microservices architecture: integration microservices
2.1 Integration Microservices with WSO2
WSO2 has come up with two ways of implementing such integration microservices.
For a configuration approach, a modern microservices-friendly integration runtime known as WSO2 Micro Integrator can be used. WSO2 Micro Integrator is a lightweight integration runtime and is the cloud-native variant of the open-source WSO2 Enterprise Integrator platform. It offers a lightweight integration runtime that natively works in the Kubernetes ecosystem. The integration logic can be authored by writing declarative syntax or graphically using drag-and-drop editors, and various Enterprise Integration Patterns can be implemented this way, while numerous other applications can be connected via connectors.
On the other hand, the programming language Ballerina can be leveraged to create the integration microservices through a code-driven approach. Ballerina is a programming language and platform that aims to fill the gap between integration products and general-purpose programming languages. The language makes it easy to create microservices that integrate and orchestrate across distributed endpoints. It attempts to provide agility and integration simplicity simultaneously; it achieves this by packaging a language, integration syntax, and environmental deployment constructs into a single code file, which is compiled into a binary and executable within a virtual machine. Since Ballerina is a programming language, developers can now apply agile methodologies to integration, contributing to faster project completion and time to market.
Integration Microservices: WSO2 Solution Pattern
Figure 3: Microservices architecture: integration microservices with Ballerina or WSO2 Micro Integrator
As shown in Figure 4, both the business logic and integration logic can be written as microservices and will reside in separate runtimes. Integration services, which integrate the microservices, can be exposed to client applications via an API gateway (or multiple micro gateways). Ballerina can be used to code these integration microservices. At the same time, WSO2 Micro Integrator can also be used to write the integration logic using the configuration style.
3. Bridging the Integration Gap with WSO2 Enterprise Integrator 7.0
WSO2 marks the introduction of a hybrid integration platform with the release of WSO2 Enterprise Integrator 7.0 because it is the only open-source hybrid integration platform to enable API-centric integration teams to connect APIs, data, streams, SaaS, and legacy apps, using either the centralized monolithic integration or cloud-native microservices architectural approaches. Current integration tools/approaches in the market often force teams toward one approach or the other — making it difficult for IT groups to agree on a single technology, vendor, or organization. With EI 7.0, developers also have the option to choose between code- or configuration-driven integration styles if they are adopting the cloud-native integration microservices architectural approach for application integration.
3.1 WSO2 EI 7.0 - Component Overview
The key components within WSO2 EI are Ballerina Integrator, Micro Integrator, and Streaming Integrator. These components are available as separate downloads. The components can seamlessly integrate with third-party brokers (such as ActiveMQ, RabbitMQ, Kafka, and NATS), workflow solutions (Camunda), and observability tools (such as ELK, Prometheus, and Jaeger/Zipkin) to provide extended functionalities.
Figure 4: EI 7.0 component overview
Let’s take a look at WSO2 EI 7.0’s key components.
Ballerina Integrator is suitable for developers who use decentralized integration architectures, microservices, and cloud-native apps; it can be used to create integration microservices using the code-first integration style.
Ballerina Integrator is based on the Ballerina programming language (ballerina.io), which is tailor-made for integration and includes first-class, cloud-native functions as well as a graphical sequence diagram tool. Ballerina incorporates fundamental concepts of distributed system integration into the language and offers a type-safe, concurrent environment to implement microservices with distributed transactions and reliable messaging. Based around the interactions of sequence diagrams, Ballerina has built-in support for common integration patterns and connectors, including distributed transactions, compensation, and circuit breakers. With first-class support for JSON and XML, Ballerina comes with key language constructs, such as service endpoints, network-aware data types, connectors, and workers, which makes it simple and effective to build robust integrations across network endpoints.
Figure 5: The Ballerina Plugin for Visual Studio Code to write Ballerina code
Ballerina Integrator can be used to create simple-to-learn, code-driven integrations. In fact, developers can directly program integration projects, helping merge the realms of development and integration, i.e., the siloed coding and waterfall-based integration teams.
Ballerina Integrator is more than the Ballerina language and includes the Ballerina installer, a sequence diagram visualization tool, key Ballerina ad-ons, pre-defined integration templates, protocol connectors (such as HTTP, File, gRPC, AMQP, NATS, Kafka, and JMS), application connectors (such as Salesforce, Amazon S3, Amazon SQS, and SAP), and extensions to optimize Ballerina for integration projects.
Micro Integrator is suitable for developers who follow either a conventional centralized ESB-like integration or decentralized integration to realize a microservices architecture through a low-code and graphical configuration-driven approach. In the centralized ESB approach, Micro Integrator can be used as the centralized ESB layer to integrate APIs, data, events, streams, and systems. Whereas in the decentralized approach, since Micro Integrator is a lightweight integration runtime and is the cloud-native variant of the open-source WSO2 ESB, it can host one or more integration microservices and run inside a Docker container. Micro Integrator offers a graphical data flow designer tool along with a configuration language based on XML to build integrations.
Figure 6: The Integration Studio plugin for Eclipse to create configuration-driven integrations
Micro Integrator supports all the WSO2 ESB features, notably a wide range of transports and protocols and the availability of 160+ connectors and adapters across various categories such as payments, CRM, ERP, social networks, or legacy systems. It can easily connect and expose data stores such as RDBMS, CSV, Excel, ODS, Cassandra, Google Spreadsheets, RDF, and any Web page as a data service. Micro Integrator provides support for all enterprise integration patterns (EIPs), database integration, event publishing, logging and auditing, validation and routes, mediates and transforms data. Data mapping can be done visually through a visual data mapper, which can map input data into output data. Even with declarative development with XML configuration, Micro Integrator also supports error handling, tracing and debugging message mediation logic. The configuration language can be extended with custom DSLs via templates and developers can use Integration Studio, which comes with rich graphical editors and other standard tools, for seamless configuration of integration logic.
Integration teams inevitably need to integrate events and streaming data, often requiring complex processing and event triggering — or worse, dealing with raw data. Most processing and triggering is difficult to configure and then connect with the core integration platform, e.g., streams carry a large volume of data moving at high velocities, and data sources/targets can be different in terms of protocols, data formats, etc.
In addition to Ballerina Integrator and Micro Integrator, EI 7.0 also offers a third integrator known as Streaming Integrator. Streaming Integrator is ideal for systems that have to process and act on streaming data. It is a cloud-native and lightweight stream processing engine based on Siddhi that understands streaming SQL queries to capture, analyze, process, and act on events in real-time. It facilitates real-time streaming data integration and analytics and can be used in both monolithic and distributed architectures. Streaming Integrator supports seamless integration with Micro Integrator and is fully compatible with major streaming messaging systems such as Kafka and NATS. It can extract data from major databases and can even integrate data in cloud storage.
Figure 7: Streaming Integrator tooling
4. Choosing an Integrator Based on an Architectural Approach and Integration Style
Unlike other products, WSO2 Enterprise Integrator 7.0 offers both code- and configuration-first options for integration teams, catering to a range of integration preferences and requirements. It also offers the capability and flexibility to design any architectural style, to opt either for centralized or decentralized solutions based on requirements. Developers can choose an integrator type of their choice based on the approaches detailed below. At the same time, Streaming Integrator can be used in both monolithic and distributed architectures.
Greenfield/Brownfield Centralized Integration
Micro Integrator is now the cloud-native version of WSO2 Enterprise Integrator — WSO2’s ESB offering — and it has all the capabilities (except for the management console). Therefore, Micro Integrator can be used as a centralized ESB suited for a centralized SOA architecture regardless of whether integration teams have to create brand new connections from one system to another (greenfield) or modify existing integrations by using a new ESB (brownfield). Integration experts or beginners, who do not yet care about microservices or serverless, would prefer the configuration approach and will find the graphical data flow designer tool offered by Micro Integrator attractive to create integrations. At the same time, Streaming Integrator can be also be used to integrate events and streams.
Greenfield Configuration-driven Decentralized Integration
Micro Integrator can serve as a runtime for integration microservices in a decentralized microservice architecture if the preferred integration style is configuration over code. Integration experts and developers, who wish to implement a decentralized greenfield microservice architecture yet prefer the configuration approach, can use Micro Integrator. Moreover, if they are looking for familiar integration tools that are inspired and influenced by existing technologies and practices that are widely practiced, proven and embraced, they will find Micro Integrator highly desirable.
Greenfield Code-driven Decentralized Integration
Ballerina Integrator can serve as a runtime for integration microservices if the preferred integration style is code over configuration. Ballerina Integrator can host integration microservices, which are written in Ballerina, a programming language that is tailor-made for integration. This will appeal to integration experts and developers, who use decentralized integration architectures, microservices, and cloud native applications, as it will enable them to directly program integration projects.
Brownfield Decentralized Integration
Most microservices projects are implemented in brownfield environments that already have legacy, cloud, and other services and, more often than not, an ESB environment. In such a scenario, a complete rewrite is usually costly. A typical approach is where newer services are implemented as microservices, creating a separate layered microservices architecture (integration microservices along with the core microservices), while the existing SOA environment (web services, APIs, legacy systems and ESB layer) remain as-is. The ESB will continue to integrate multiple services and other systems and will be used as the centralized bus that connects all these services and systems outside the MSA. The integrator type to host the integration microservices can be chosen based on the preferred integration approach — Micro Integrator for configuration-driven integration and Ballerina Integrator for code-driven integration.
Figure 8: Decentralized microservices architecture and SOA in coexistence
As shown in the architecture above, a layered MSA coexists with the SOA. New functionality can be implemented within the MSA and the existing services and applications in the SOA can reside as-is.
The new WSO2 Enterprise Integrator 7.0 is the only open-source hybrid integration platform that enables API-centric integration teams to connect APIs, data, streams, SaaS, and legacy apps using any architectural integration style (e.g., centralized/monolithic integration or cloud-native microservices integration).
EI 7.0 is also the first solution to use Ballerina, a revolutionary new programming language for programming network-distributed applications. For the first time, developers can now apply agile methodologies to integration.
For integration professionals who prefer graphical or configuration-driven integration with no code, EI 7.0 offers Micro Integrator. It has the same battle-tested runtime used in previous versions but is optimized for both cloud-native and centralized ESB style architectures.
Micro Integrator and Ballerina Integrator address two different integration approaches (configuration-driven vs. code-driven) and an integration user can decide which approach is best.
For any integration use case that requires processing a stream of events, EI 7 facilitates it with a lightweight stream processor that understands streaming SQL queries to capture, analyze, process, and act on events in real-time.
For more details about our solutions or to discuss a specific requirement contact us.