Brownfield Integration: Why It's Important For Modernizing Your Enterprise

Transforming a conventional business into a digital business is essential to expand the business and increase its productivity. Integrating heterogeneous systems and building an ecosystem with these integrated components are fundamental requirements in the digital transformation of enterprises. Many organizations have one or more legacy systems that provide limited support for integrating with other systems. However, it is impractical to replace all existing systems by building them from scratch. Therefore, the challenge is to drive all these existing systems towards brownfield integration. This article delves into the importance of brownfield integration, challenges you may encounter, and how you can use WSO2 Enterprise Integrator for brownfield integration.

Brownfield vs. Greenfield

Brownfield land (image on the top) is an area of land that has been used for commercial or industrial purposes. Greenfield land (image on the bottom) is land that has never been used for such industrial purposes. Similarly, in software, a brownfield is a system that is already in existence. Brownfield integration means integrating existing systems with the rest of the world. A greenfield system could be a system that is developed for a completely new environment without taking into considerations how to integrate with other systems in existence. Brownfield land could contain hazardous substances and construction on such land has to be handled carefully. Similarly, integration with brownfield software systems has to be done with extra care compared to building a system from scratch.

Why is Brownfield Integration Important?

Most modern systems support integration with other systems through APIs that are exposed through well-known protocols and standards. However, it's difficult to expect all existing systems of an organization to be capable of integrating with other systems with well-known standards and APIs.

Most legacy systems use proprietary or non-standard ways to talk with the outside world. These legacy systems will be eventually replaced by modern systems, even though it takes time. Nevertheless, organizations may want to accelerate their digital transformation strategy rather than delay this process by waiting to replace the existing system with a modern one. Building everything from scratch is not cost effective and is also a tough call for a manager to make, especially if a system plays a vital role in the core business of an organization . Furthermore, no one wants massive change at once. Therefore, typically a part of the new system is integrated with the existing system and both co-exist during a transition period. This gives decision makers more freedom to change the system in its entirety if needed and minimize the business risk to the organization. Not all existing systems are replaceable but this should not be a reason for not digitally transforming an enterprise. If any existing system is not replaceable, then we need to find new ways to bring functionality to these systems by integrating them with other systems. Driving these existing systems towards brownfield integration is an important aspect in today’s digital world.

Let us now look at the various types of systems in brownfield integration.

Types of Systems in a Brownfield Integration Environment

Depending on integration capabilities, modern enterprise systems can be categorized into four types.

1. Systems based on standard protocols

Some existing systems may have a way to integrate with other systems in a standard way. (A standard way of integration means support for a standard protocol like HTTP, JMS, FTP, etc). It’s highly likely that a system built in the past decade contains APIs for integration. Most integration solutions support this type of integration and it’s comparatively simple to integrate with these systems.

2. Systems based on non-standard or proprietary protocols

Some systems provide extension points to integrate with other systems. However, these extension points are not based on a common standard. For example, some systems provide SDKs to build programs to integrate with their systems. Proprietary systems like SAP do the same. A majority of brownfield systems belongs to this category. Compared to systems based on standard protocols, integration with these systems is not straight forward, but also not impossible.

3. File-based systems

Most old systems use files, like CSV, to transfer data. Integration with these systems therefore needs files.

4. Systems that don’t provide any integration points

Some standalone systems do not provide any extension points for integration at all. A majority of such systems uses databases to store data. Integrating with these systems require integrating with their databases.

Key Functionality Requirements

Now let’s look at the features required in an integration solution for brownfield integration. These features include:

• Message transformation
• Message routing
• Service orchestration
• Protocol switching
• Parallel processing

Let's examine each of these in detail.

Message Transformation

In this illustration, we have the brownfield on the right. On the left, we have the new system we are building. Imagine that we get a request from the user and we can’t send the message as it is to the existing system. In such a case, we need to transform and construct the message as required by the existing service.

Message Routing

In the picture, the client is sending two types of messages. Routing can be based on the content of the message, a header, or some property of the message. Depending on the type of message, it needs to be routed to the correct service.

Service Orchestration

When building an integration solution we might have to connect with multiple systems and orchestrate them to build a new service. This picture shows 3 systems that exist in the brownfield. In this scenario, we orchestrate them to build a system that provides new functionality and invoke services one after the other sequentially.

Protocol Switching

In the above illustration, the brownfield system is exposed over JMS protocol and we need to build a new system that is exposed over HTTP protocol. The ability to perform protocol switching is a key feature for an integration solution.

Parallel Processing

Parallel processing is the other key functionality. In this scenario, we have two systems and we need to talk to both systems at the same time to reduce the latency. The integration solution clones the incoming request message and sends it to both services simultaneously.

WSO2 Enterprise Integrator has the capability to perform brownfield integration and let’s now take an in-depth look at how it works.

Implementation of Key Functionality Requirements with WSO2 Enterprise Integrator

WSO2 Enterprise Integrator is an open source, hybrid integration platform that allow developers quick, CI/CD-friendly integration of any application, data, microservice or system. It provides an enterprise service bus, which mediates and transforms data between legacy systems, SaaS applications, services, and APIs. You can employ APIs to interface with legacy systems while insulating the underlying complexities securely and cost-effectively. WSO2 Enterprise Integrator has industry ready connectors and templates, and allows you to develop your own custom connectors for corner cases.

This integration platform covers all angles of enterprise integration - service integration, data integration, business processors, and messaging.

We can implement key functionality requirements for brownfield integration with WSO2 Enterprise Integrator and each requirement is discussed in-depth in the following sections.

Transformation

Let’s revisit this picture and place WSO2 Enterprise Integrator between the client and the server. WSO2 Enterprise Integrator is capable of transforming messages from any one format to any other format. It has built-in support for common message types like JSON, XML, CSV, etc. and we can extend the functionality to handle any type of message format that is not supported by default. The unit of message processing is called a mediator in WSO2 Enterprise Integrator, which has several mediators for different types of transformations. For simple message enriching, there is a mediator called enrich mediator. For simple message constructions, there is something called payload factory mediator, which allows to define the expected payload format and parameterize whichever element we want. The Datamapper mediator allows us to visually map data fields in the input message and the output message. For advanced transformations, we can even use an XSLT script and plug it using the XSLT mediator.

Routing

WSO2 Enterprise Integrator supports various kinds of routing requirements. Routing can be based on the content of the message, the header of the message, or on some property of the message. This integration platform also has mediators like Switch for message routing, which is very similar to the Switch statement in a programming language and there is also a mediator called Filter, which is similar to an If-else statement.

Orchestration

WSO2 Enterprise Integrator has a simple mechanism to orchestrate services. With the use of Call mediator, we can call services in a sequential manner. An asynchronous non-blocking transport implementation is used for invoking an HTTP service using the Call mediator. Even though the user sees a sequential synchronous invocation, the underlying threading mechanism is asynchronous. These threads are non-blocking and do not wait for the backend to respond. Hence, it will help to increase the productivity while utilizing fewer resources.

Protocol Switching

WSO2 Enterprise Integrator has the capability to receive and deliver a message over any protocol. The message is received through a transport listener or an inbound endpoint and sent through an endpoint using a transport sender. It has built-in support for almost all standard protocols and some well known proprietary protocols. There's flexibility to extend and introduce new transport listeners and senders to handle any unknown protocols.

Now let’s look at how we can integrate with different types of systems in the brownfield.

Brownfield Integration with Different Types of Systems

Integration with systems that talk in standard protocols

WSO2 Enterprise Integrator has built-in support for a wide variety of standard protocols such as the ones listed below and many more:

• HTTP/ HTTPS
• JMS, AMQP, MQTT, KAFKA message protocols
• FTP/ SFTP/ FTPS/ SMB file handling protocols
• SMTP/IMAP mail protocols

In this diagram, there is an existing legacy system that is capable of integration over the FTP protocol on the right. We need to expose this as an API to the outside world over the HTTP protocol. We place WSO2 Enterprise Integrator in the middle and connect to the existing system over the FTP protocol. We can then expose a new API to the outside world over the HTTP protocol in WSO2 Enterprise Integrator.

Integration with systems that talk in non-standard/proprietary protocols.

WSO2 Enterprise Integrator has support for some well known proprietary systems like SAP. But you could come across systems which WSO2 Enterprise Integrator doesn’t offer support out of the box. For these, we can easily extend the functionality of WSO2 Enterprise Integrator to support these proprietary or legacy systems. There are various extension points available in each layer of WSO2 Enterprise Integrator's architecture to extend its functionality.

The above illustration shows a legacy system which talks in a nonstandard protocol on the right. We need to expose this as an API to the outside world over the HTTP protocol. We place WSO2 Enterprise Integrator in the middle but it does not know how to talk to the existing system. Therefore, we must write a connector and plug it into WSO2 Enterprise Integrator. We can then connect with the existing system and expose an API over the HTTP protocol.

Integration with File-based Systems

As discussed earlier, many legacy systems use files as the medium to transfer messages between systems. These files are transferred over various protocols. It can be a local file system or remote file systems based on FTP, SFTP, etc. WSO2 Enterprise Integrator supports all kinds of file transfer protocols. There are different kinds of file handling use cases. One involves polling a directory or location periodically, selecting files, and processing them. For listening and polling to a directory, WSO2 Enterprise Integrator has a File Inbound Endpoint and a VFS Transport Listener. Another use case involves looking for a file in a particular location, upon some external trigger. For example, if we receive a request for an API, we select a file from a directory. For this type of integration, WSO2 Enterprise Integrator has a connector called the File Connector. Yet another use case is writing data to the file system. Both the File Connector and VFS Transport sender are capable of doing this.

The diagram below illustrates the first use case we discussed, where the legacy system writes data to the file system. WSO2 Enterprise Integrator listens to that directory, selects the file, and performs a web API call using the data in the file.

In this diagram, WSO2 Enterprise Integrator exposes an HTTP API to the outside world. When it receives a request from the client, data contained in the HTTP request is written as a file into the file system. The legacy system can then process this file.

Integration with Systems that Use Databases

As mentioned earlier, some standalone systems do not provide any extension points for integration at all and store their data in databases, and the way to integrate with these systems is by integrating with their databases. To expose databases and other data sources as services, WSO2 Enterprise Integrator provides a rich set of data integration features.

The diagram above shows a simple scenario where WSO2 Enterprise Integrator is connected to a database of an existing legacy system and the data available in these databases are exposed over an HTTP API using WSO2 Enterprise Integrator.

Summary

In this article, we discussed why brownfield Integration is important and how WSO2 Enterprise Integrator can be used for this purpose. We looked at key functionality areas of brownfield integration and the features required by an integration solution to do brownfield integration. As discussed, WSO2 Enterprise Integrator can fulfill all the key integration requirements covering all the aspects of integration - service integration, data integration, business processing, and messaging. Learn more about its capabilities here.