Deployment Patterns in WSO2 Enterprise Integrator

Introduction

WSO2 Enterprise Integrator is a comprehensive integration solution with a plethora of integration capabilities packaged into a single product in the form of different profiles. Thus it provides the flexibility of architecting these profiles for real-world deployments of WSO2 Enterprise Integrator depending on the requirement. A deployment pattern is a commonly used configuration of the profiles of WSO2 Enterprise Integrator and there are several reusable deployment patterns that are used by organizations to host different integration solutions. This article discusses how seven of these deployment patterns can be applied in an enterprise to achieve different integration requirements.

Different Profiles of WSO2 Enterprise Integrator

The integration solution requirements of different organizations, teams, and individuals can vary greatly depending on many factors like the application domain, user demand, transactional requirements, etc. WSO2 Enterprise Integrator provides a collection of different profiles with different business capabilities such as mediation, messaging, and analytics. This enables solution architects to select only the profiles they need to implement their integration solutions. There can be integration solutions that only need the integrator profile and there can also be advanced integration solutions that need multiple profiles. Additionally, Integration engineers also have the capability to scale different profiles independently depending on the application demand and to tune the instances for different resource utilization depending on the profile.

Following is the list of profiles available in the enterprise integrator with different capabilities:

• Integrator: For system and data integration
• Business Process: For long-running business processes implemented on BPEL, BPMN, and Human Tasks
• Message Broker: For message brokering requirements
• Micro-Integrator: For writing container friendly composite services
• Analytics: For monitoring integration solutions

Deployment Patterns with WSO2 Enterprise Integrator

Pattern 1 - Enterprise Integration (Systems + Data)

Pure enterprise integration is the smallest/most basic deployment of the conventional deployment patterns. What needs to be integrated in such a deployment could fall under 3 categories:

1. Legacy applications: In-house services that are built and maintained by the organization itself to cater to specific requirements.
2. Saas applications: Software as a service applications that are provided by third parties. E.g., Salesforce, SAP, etc.
3. Data sources: Data sources that contain information about the organization (customers, products, etc.) and could be any existing relational databases, no-SQL databases or even spreadsheets containing data.

A system that enables this type of integration should have the following capabilities:

• Message routing: Routing messages from one service to another and then from/to datasource back and forth in order to complete a single business functionality.
• Protocol switching: A request might have to be forwarded to many services that use different protocols such as HTTP/S, FTP, SMTP, etc. Thus the integrating system should be capable of switching protocols accordingly.
• Message transformation: Due to the heterogeneous nature of the applications that are being integrated, message formats might need to be changed internally to enable communication. One such example is when REST services need to be integrated with SOAP services.
• Service orchestration: This comes into play when legacy backends/ Saas APIs need to be exposed and simple service endpoints.

The following diagram summarizes all the above requirements along with the capability to monitor and analyze the flow of requests/responses.

The Integrator profile of WSO2 Enterprise Integrator is the solution recommended by WSO2 for this minimal deployment. Cloud connectors, endpoints, and datasources are the inbuilt features that are provided out of the box by the Integrator profile of WSO2 to make the above integration possible. The analytics profile of the enterprise integrator could be useful if/when monitoring is required. The incoming request count, the request rate, and the message latency would be prominent among the statistics that could be monitored through WSO2 Enterprise Integratoranalytics. The simple deployment diagram consisting of all components can be seen below.

Even though a single instance of each of these profiles (integrator profile and the analytics profile) is sufficient for the integration, it is recommended to configure two instances from each. This is to ensure high availability when considered as a complete solution, especially in the production environment.

Pattern 2 - Enterprise Integration with Message Reliability

The deployment pattern of Integration with Message Reliability comes into play when there is a requirement for messaging capabilities in addition to the above integration requirements in an organization. Message persistence, guaranteed delivery, asynchronous publishing, and subscribing, etc. could be among the basic messaging capabilities that are expected from the solution. The following diagram shows all the requirements in such a scenario with the addition of the requirement of a basic integration.

The broker profile of WSO2 Enterprise Integrator is specifically designed by WSO2 to cater to this requirement. The following diagram shows how the broker profile could be embedded into the deployment.

The broker profile is configured to be integrated with the integrator profile in order to bring in the messaging capabilities. The broker profile itself is not capable of communicating with SaaS applications, legacy apps or datasource hence the integrator profile should be used as same as in the conventional integration. Incoming requests from SaaS applications, inhouse apps or data sources should be forwarded to the broker by the integrator and then consumed back for the continuation of the message flow. The message broker profile will not publish any statistics related to messaging hence the analytics made available through the analytics profile are the statistics published through the integrator profile. ‘Carbon metrics’ is used within the broker profile for monitoring requests.

Pattern 3 - Enterprise Integration with Business Processes

One of the most common use cases of organizations is to have stateful processes integrated with other services which are stateless, where, among the stateful processes there could be flows that require human intervention. All requirements of such an integration as a whole can be seen in the following diagram.

The business process profile of WSO2 Enterprise Integrator is specifically defined for modeling business processes which are such stateful services. It has the inbuilt supports for both BPMN (Business Process Model and Notation), which is a graphical representation for specifying business processes and for BPEL (Business Process Execution Language), which is an XML-based language used to define enterprise business processes within Web services. Also, it brings in the support for human intervention through human tasks, which can be integrated with BPEL and also be used in isolation. The following diagram explains how the business process server could be integrated with the integrator profile of WSO2 Enterprise Integratorin order to model and deploy and execute business process flows in an organization.

The business process profile of the integrator brings in the capability to augment the common integration with business flows. Even though it is possible to connect in house applications to the business process profile directly, the most common use case is to integrate the services with the BPS profile using the integrator profile in the middle. It is also mandatory to use the integrator profile when it is required to connect cloud services with other services.

Pattern 4 - Modern Enterprise Integration

In large organizations, shared integration platforms are built to host multiple integrations. Hence, it is common for such organizations to require the full scope of integration capabilities in the platform. This pattern describes how an organization can deploy all major profiles together in a single environment.

At a very high-level, the interactions between different internal and external components will be similar to the above diagram. There will be Saas applications, multiple datasources, In-house/Legacy application, and the microservices in a typical enterprise that need to be integrated using the platform which is being built. WSO2 Enterprise integrator comprises of all such service integration, data integration, business process, and brokering capabilities. Therefore, integration between these heterogeneous systems is quite simple and easy.

As in previous patterns, the integrator profile acts as the central component which is used to integrate everything while the message broker and the business process profiles augment the integration capabilities in the platform. In addition, where microservices are used in the enterprise, the message broker profile can be used in between the microservices and the integrator in order to communicate asynchronously. Finally, the analytics profile can be configured with the integrator profile, message broker profile, and business process profile for monitoring and analytics purposes.

Pattern 5 - Integration on Cloud

Although it is quite possible to have all the combinations of the aforementioned deployment patterns, the majority of the organizations still possess the requirement of performing pure integrations of systems and data thus making the first deployment pattern the most popular deployment pattern out of all. Even the smallest integration could span up to multiple nodes depending on the requirement to have an HA deployment and monitoring aspects. Therefore, managing the deployment becomes a tedious task sometimes. The integration cloud managed by WSO2 takes this burden away from organizations by bearing their deployment responsibilities from single-node deployments to multi-node large scale deployments.

The connectivity between on-premise systems and integrator nodes is mandatory in most of the deployments. As depicted in the above diagram, the integration cloud allows integrator nodes to connect to on-premise systems using virtual private networks (VPN) and cloud systems using the internet. One of the key points to be noted is that the integration cloud allows monitoring of the integration using tenant isolation.

Pattern 6 - Hybrid Integration

A typical enterprise contains many on-premise systems that are very hard or impossible to be moved to the public cloud (Google cloud, Amazon cloud, etc.) due to many factors such as the underlying technology, security, and organization/government policies and so on. When the cost of maintaining an on-premise system and the benefit of hosting in public cloud are considered, sometimes it is best to move the systems which can be moved to the public cloud. This is true for integration solutions as well. Since there are multiple profiles in WSO2 Enterprise Integrator, organizations have the flexibility to deploy different profiles either in public cloud or on-premise. The only requirement is to assure that these profiles can communicate with each other securely. The recommended mechanism for this communication is to configure a VPN between the public cloud and on-premise so that they can communicate securely.

WSO2 Integration Cloud is the iPaaS solution that can be used to host the integrator related artifacts in the cloud. This pattern describes using the integration cloud and the WSO2 Enterprise Integrator profiles hosted on-premise together to implement an integration solution.

This is also very similar to pattern 5 where the integration artifacts are deployed in the public integration cloud. A VPN is used to communicate between the integration cloud and on-premise systems. Additional components are the message broker profile and the business process profile which are deployed on-premise.

As depicted in the diagram, a two-node cluster is configured for both message broker and business process profile to ensure high availability. No additional configurations are required to ensure the high availability of the integrations hosted in the cloud since that is taken care of by the WSO2 integration cloud team.

Pattern 7 - Containerized Deployment

There are mainly two types of flavors of how software development can be done; Centralized integration and Decentralized integration. One perspective is that centralized integration is chosen when governance is more important and decentralized integration is chosen when agility is more important. The integration problem pertains and is prominent even in the microservices architecture, when a complete solution is being built, especially since the applications are now broken into a collection of services and those services need to be integrated. Developers get more freedom in developing the domain services when the integration is divided into a separate microservice since they do not have to worry about the parameters like protocol and technology stack. This pattern discusses how it can be achieved using the Micro Integrator profile available in WSO2 Enterprise Integrator in a containerized environment.

Integration artifacts for the micro integrator profile can be developed similar to how they are developed for the integrator profile. All service and data integration capabilities that the integrator profile possesses are also available in the micro integrator profile. It is also possible to create a docker image for composite services with the integration artifacts using the micro integrator profile. This docker image could then be used to deploy the service in the containerized environment.

Micro integrator does not require any database configuration since it, by default, uses a file-based registry instead of a DB backed registry. Therefore, a datasource configuration is not required except for data integration scenarios. For monitoring and analytics purposes, an HA analytics cluster can be configured where each micro integrator pushes data to these analytics profile instances. Since the analytics profile uses a DB backed registry underneath, it is required to have the configurations to connect to an external database.

Summary

WSO2 Enterprise Integrator is comprised of a number of profiles that can be interconnected in different arrangements to resolve different integration problems. A deployment pattern is one such configuration of profiles that is identified as commonly used among organizations. Following are the deployment patterns that are described throughout this article:

1. Enterprise Integration (Systems + Data): The most basic pattern that can be used for service and data integration use cases.
2. Enterprise Integration with Message Reliability: Utilizes the message broker profile to provide message reliability.
3. Enterprise Integration with Business Processes: Used when the use cases need stateful integrations in addition to stateless integrations.
4. Modern Enterprise Integration: Used in large organizations where all the business capabilities (integration, messaging, business process and analytics) of the enterprise integrator is utilized.
5. Integration on Cloud: Used when the organization does not want to manage the middleware deployment themselves.
6. Hybrid Integration: Some organizations host part of the middleware runtimes on-prem while using the integration cloud for service and data integrations for various reasons like performance. This pattern is targeted for those use cases.
7. Containerized Deployment: Used when organizations want to host the integration middleware also in a container based environment like Kubernetes.

The simplest deployment pattern consists of the Integrator profile alone for data and service integration whereas the most common deployment pattern includes all profiles of WSO2 Enterprise Integrator - the Integrator, Micro-Integrator, Message Broker, Business Process, and Analytics profiles.