Synchronizing Sales Data with Salesforce, EDIFACT B2B Messages, and Ballerina

Businesses often exchange a variety of documents like invoices, purchase orders, insurance claims, and shipment handling orders in their B2B communications. Electronic Data Interchange (EDI) messages, primarily using standards like EDIFACT and X12, play a significant role in these exchanges. To provide a seamless digital experience for customers and partners, organizations need to integrate these EDI-based B2B channels with their internal IT systems.

Customer Relationship Management (CRM) systems are also crucial in an organization's IT landscape for managing sales and customer data. Therefore, integrating CRM with all customer touchpoints is essential for capturing up-to-date information. Given the frequent usage of EDI-based B2B communication in business, companies dealing with enterprise customers can benefit greatly from effectively integrating their CRM systems with B2B channels.

Ballerina's EDI package provides a straightforward yet robust solution for managing EDI messages, with built-in support for EDIFACT and X12 standards. Moreover, Ballerina offers connectors for widely used systems like Salesforce, Oracle NetSuite, and Microsoft Dynamics 365. In this article, we'll explore how Ballerina's EDI capabilities, connectors, and data processing features can be used to streamline the integration between CRM systems and B2B channels, with a focus on Salesforce and EDIFACT B2B messages.

Architecture Overview:

It's important to note that besides CRM systems such as Salesforce, various other systems, including Enterprise Resource Planning (ERP), auditing, and warehouse management systems, require access to B2B messages. In such situations, messaging systems like Kafka can be employed to create an event-driven architecture. This architecture allows multiple consumers to process incoming and outgoing messages. The diagram below illustrates an architecture that integrates B2B channels with Salesforce and other IT systems using Kafka and Ballerina. It highlights an example scenario involving quote requests and responses.

Figure 1: Architecture overview

In the architecture mentioned above, we use an FTP server for exchanging EDI messages. However, alternative methods like HTTP, Amazon S3, Azure Blob Storage, or SFTP can also be employed. Ballerina components act as the glue between source systems, target systems, and message brokers. These Ballerina components take care of protocol conversion, message transformations, and other tasks related to EDI data. In addition to the Ballerina components demonstrated here, any number of additional software components can subscribe to relevant topics in the message broker to consume B2B messages.

For each EDI message type, we use a Ballerina component and a Kafka topic. This means you can easily extend this architecture to handle new message types without affecting existing components.

A Brief Look at EDIFACT

Before we delve into a specific example using this architecture, it's essential to understand the EDIFACT standard we'll be working with. EDIFACT is one of the most widely used EDI standards, particularly in Europe and countries outside North America. It defines over 230 different messages, known as transaction sets, covering various industries like retail, manufacturing, logistics, and finance.

For instance, the "QUOTES" message captures comprehensive details for quotations, including customer identification, quotation dates, payment terms, and product details. The specification goes into great detail; under payment instructions, there are standardized codes for more than 60 methods, such as cash (10), cheque (20), and credit transfer (30), ensuring clear and efficient transactions.

Example Scenario

Now, let's focus on an example scenario within an organization called TechPulse, involving the processing of quote requests and responses:

• Customers in need of quotations for specific products place quote request messages (EDIFACT REQOTE) in a designated FTP location.
• These EDI quote request messages are transformed into an internal JSON format and placed in a Kafka topic.
• JSON quote request messages are used to create opportunities in Salesforce.
• Once TechPulse issues quotations, they are placed in another Kafka topic as JSON messages.
• Relevant Salesforce opportunities are updated with data from these quote response JSON messages.
• JSON-based quote response messages are converted into EDI messages (EDIFACT QUOTES) and placed in the output FTP location.

The following sections will walk you through the Ballerina components responsible for these tasks. Note that each Ballerina component is designed to run periodically (e.g., using a cron job).

Reading EDIFACT REQOTE Messages and Publishing Transformed Messages to Kafka

The "edi-to-quote-request" component reads from the designated FTP location and translates incoming EDIFACT REQOTE messages into an internal format. These messages are then published to a Kafka topic, making them accessible to any authorized system. Below, you can see the source code of this component. It's worth noting that one of Ballerina's built-in EDIFACT packages (ballerinax/edifact.d03a.retail) is used for processing the REQOTE message.

Ballerina can visualize code as sequence diagrams that illustrate connections to external systems and control-flow constructs. Visualization of the above code is shown below:

Figure 2: Code visualization

Creating Salesforce Opportunities for Quote Requests

The "quote-request-to-salesforce" component takes these internal messages, creates Salesforce opportunities, and includes the appropriate products within these opportunities. At this stage, the prices are obtained from Salesforce's price book.

Updating Salesforce with Quote Response Data

To ensure that Salesforce stays aligned with the quotation process, the opportunity initially created for the related quote request must be updated with the specifics of the quote response. This task is handled by the "quote-response-to-salesforce" component, which promotes the opportunity stage to "Proposal/Quote" and updates the prices of all ordered items based on the prices provided in the quote.

Sending transformed EDIFACT QUOTES messages to an FTP location

In the final step, the "quote-response-to-edi" component transforms internal quote-response messages into EDIFACT QUOTES messages. These messages are subsequently stored in an FTP location for access by the relevant business partners.

You can find the entire source code for this example solution on GitHub.

How it Works with a Sample Quotation Request

Once the solution is up and running, you can deposit quote request EDIFACT messages into the FTP input location. Here's a sample EDIFACT REQOTE message:

Once you've executed the "edi-to-quote-request" component, the EDI quote request will undergo transformation into an internal JSON format. This transformed message will then be published to the "quote-requests" topic. Below, you can see the JSON message that results from the conversion of the initial request:

Now once the quote-request-to-salesforce component is executed, a Salesforce opportunity will be created for the above message as follows:

Figure 3: Salesforce opportunity

Products associated with the newly created Salesforce opportunity are shown below:

Figure 4: Salesforce opportunity

Suppose TechPulse chooses to offer a 5% discount on the first product, which is the VisionPro TV. Additionally, let's assume that TechPulse has a limited stock of only 50 units for the second product, the NexaView TV. These decisions lead to the following quote response JSON message, which will then be stored in the "quote-responses" topic (you can generate a sample quote response by running the "quote-processor" component).

Now, once the quote-response-to-salesforce component is executed, it picks this quote response and updates the stage of the opportunity created for the corresponding quote-request as follows:

Figure 5: Quote response to Salesforce

It also updates the product list associated with the opportunity by updating quantities and discounts as shown below:

Figure 6: Quote response to EDI

Once you run the "quote-response-to-edi" component, it takes the quote-response message and transforms it into the EDIFACT QUOTES format. This converted message is then stored in the output FTP location.

In this architecture, we've utilized four distinct Ballerina components, each designed to handle specific tasks, following microservices architecture principles. However, the granularity of these components can vary depending on factors such as system load, extensibility requirements, infrastructure costs, and team structure.

For instance, in scenarios with budget constraints and low load, it's possible to consolidate all tasks into a single Ballerina component. On the other hand, if the system needs to support multiple transport types like HTTP, Amazon S3, and FTP, it may be necessary to have a dedicated component for each transport type while isolating the EDIFACT-to-internal format transformations into another component.

In terms of deployment, each Ballerina component can be packaged as a Kubernetes pod, making it compatible with various container orchestration systems such as plain Kubernetes, OpenShift, Amazon EKS, or Azure AKS. Alternatively, this architecture can be implemented on an internal developer platform like Choreo. This approach expedites the go-live process by providing pre-configured CI/CD pipelines, multiple deployment environments (e.g., development, testing, staging, production), enhanced security features, and built-in analytics.