Business-to-Business Integration with Ballerina

This article is based on Ballerina Swan Lake Update 6 (2201.6.0).

Introduction

Imagine a scenario where Voltex, an electronics appliance manufacturer, supplies its products to retailers like TechTown. Voltex depends heavily on suppliers for parts. One day, TechTown launches a significant promotion for Voltex's products. However, outdated manual ordering systems prevent TechTown from placing timely orders. When Voltex receives late orders, their own manual systems slow them down, and they can't quickly arrange more supplies. This coordination failure triggers a devastating supply-demand mismatch, causing order cancellations, customer dissatisfaction, and a tarnished reputation.

This is an example scenario that could happen if proper integrations among business partners are not implemented.

B2B Integrations

Integrations can be categorized into three types: internal, business-to-consumer (B2C), and business-to-business (B2B).

• Internal integrations are the integrations within an organization. It involves varied standards, often connected via an internal integration layer (e.g., service bus). Data exchanges are relatively flexible and facilitated by collaborations among developers.
• B2C integrations are the integrations between an organization and consumers of its products. Organizations generally control the interface design (APIs) and data exposure in B2C communications.
• B2B integrations are the integrations between an organization and its business partners. These integrations necessitate stricter adherence to data format standards and stringent controls over the data exchanged. Alterations are infrequent, necessitating inter-organizational coordination.

B2B Integrations and Digital Transformation

As illustrated in the example in the beginning of this article, B2B integrations and timely orchestrations across partner systems play a key role in digital transformation initiatives. Let's clarify this further with an example from the Retail Domain.

Consider a retail store with a web-based shopping portal and a backend with a REST API. This is a B2C integration, as it facilitates the retail store to interact with consumers. The retail store may also have internal systems for sales, procurement, inventory, etc. These systems need to communicate with each other, which are internal integrations. Finally, the retail store needs to communicate with manufacturers to order goods. This is a B2B integration, as the retail store's procurement system connects with the manufacturers' sales services to place orders.

Image 1: Internal, B2C, and B2B integrations in retail domain

Why don't we use JSON and XML for B2B integrations?

Although JSON and XML are popular data formats for internal and B2C integrations, there are a few reasons why they are not widely used for B2B integrations:

• Consensus: B2B data formats and schemas need to be agreed upon by several companies. This can be difficult to achieve, especially for large and complex integrations.
• Complexity: A single B2B message can contain a vast amount of data, which requires complex schema definitions. This is usually labor-intensive to develop and maintain.
• Legacy systems: Pre-JSON/XML era standards are widely used for B2B integrations. It is unlikely that new schema sets will gain acceptance across organizations.

Electronic Data Interchange (EDI)

EDI is the de facto standard for B2B integrations. It is heavily adopted by organizations across multiple industries, and in most scenarios, the only method for integrating with business-partners.

Here is an example EDI message:

Each line in the EDI message is called a segment. A segment can have multiple fields, each containing a data value or multiple sub-fields (called components). Multiple segments can optionally be grouped into segment groups, which are hierarchical.

The above message starts with a header segment (HDR), specifying it as order number 1201 from ABC_Store dated 2008–01–01. The subsequent segments (ITM) are the individual items within the order. Each ITM segment lists a product code (e.g., A-250, A-45, D-10) and the corresponding quantities ordered (e.g., 12, 100, 58). The tilde (~) marks the end of each segment.

In reality, EDI message structures are far more complex than the above example and can contain numerous types of segments and segment groups. This complexity, along with the need for agreement among organizations, has led to the standardization of EDI message types under various specifications like X12 and EDIFACT. For instance, X12 has established standards for over 350 B2B message types. These include an invoice message (EDI 810), purchase order (EDI 850), payment order (EDI 820), shipment notice (EDI 856), and insurance claim (EDI 837), among others.

B2B integration architecture

A B2B gateway is typically used to bridge B2B and internal communications. The gateway receives EDI messages from partners, converts them to internal formats, and then routes them to appropriate internal systems.

The gateway also converts internal messages to EDI format and sends them to partners.

Image 2: B2B gateway architecture

In the architecture described above, all external communications pass through the partner management and authentication module. This module is responsible for handling partner certificates and verifying incoming messages using identifiers like partner IDs and digital signatures. Each trading partner has its dedicated module that transforms partner-specific EDI formats into a standardized internal format. The EDI processing and transformation module performs various tasks, such as validation, filtering, and enrichment on EDI messages, and converts them into relevant internal formats like JSON and XML. This module also sends both the EDI data and associated metadata to the message persistence and EDI tracking module, which further processes and stores them in a data store for auditing, replaying, and monitoring purposes.

When it comes to implementing this system, the components mentioned earlier can be deployed as separate runtime components, especially in microservices environments. Additionally, the B2B gateway functionality can also be integrated into the application's logic itself, depending on the technology stack being used.

EDI message processing with Ballerina

Now, let's delve into the actual implementation process. Among the various options available, in this article, we have chosen the Ballerina language for the implementation. We made this choice primarily because Ballerina offers built-in support for handling EDI, and it's well-suited for developing data-oriented business applications. Ballerina simplifies the handling of complex EDI messages by converting them into Ballerina records and vice versa, based on a specified schema. Once converted, EDI data can be manipulated just like any other record, enabling a wide range of data processing operations. For example, users can validate EDI fields against organization-specific criteria, extract relevant fields to create new records, save selected data elements into a database, or route data based on EDI field values.

Ballerina uses a JSON-based schema to describe EDI messages as shown in the following example:

This schema has two main parts: the header segment and the itemDetails segment group. The header segment includes essential fields like code, orderId, organization, and an optional date field. On the other hand, the itemDetails segment group can appear multiple times and consists of two segments: item and supplier.

The item segment needs code, item, and quantity (with an integer data type) fields. Meanwhile, the supplier segment requires code, supplierCode, and an optional promotionCode fields.

Here's an example of an EDI message following this schema:

To work with this EDI schema in Ballerina and process EDI messages, you can follow these steps:

1. Obtain the Ballerina EDI tool by running the following command:bal tool pull edi

2. Create a new Ballerina package with the following command: bal new order_processing

3. Navigate to the "order_processing" folder, and use the command below to generate Ballerina code for the given schema: bal edi codegen -s resources/schema.json -o sales_order.bal

Please note that the command assumes the schema is located in the "resources" folder, but it can be placed anywhere in the file system.

This process will create a file that includes all the necessary records for representing data in sales order EDI messages. Additionally, it contains two functions for converting EDI messages to records and vice versa. 

4. Use the generated code to load EDI messages into Ballerina records and apply any operation on those records.

Ballerina program for printing the order Id, customer organization name and total number of ordered items from an EDI message

Ballerina's simplified EDI handling capabilities and flexibility in deployment empower developers to implement EDI processing in the way that best suits their needs. For instance, you can use Ballerina to create a scalable application, serving as a centralized B2B gateway. Alternatively, you can opt for implementing separate Ballerina microservices to manage various EDI tasks like handling message variations, converting EDI to internal formats, validation, transformation, and persistence. These microservices can be deployed as cloud-native applications, for example, using Choreo. On the other end of the spectrum, especially for simple applications, you have the option to integrate EDI processing steps directly into your application's logic.