Batch Processing with WSO2 ESB - An Overview

This article demonstrates the batch processing capabilities of WSO2 Enterprise Service Bus (ESB).

Pre-requisite

WSO2 ESB

Introduction

Most legacy applications produce batch files that contain hundreds of entries, which need to be processed by other applications as individual messages. This brings about a need to introduce an integration layer that's capable of processing these batch data and transforming them to messages that can be consumed by other applications in the organization’s eco-system. This article looks at how the WSO2 ESB can be used to process batch data and transform them to individually consumable messages that can be sent to other applications in the organization.

Applies to

Table of contents

1. Processing batch data
2. WSO2 ESB to process batch data
3. The Scenario
4. Summary

Processing batch data

A batch file comprises multiple records that need to be processed before they can be sent to other systems in the organization. This process would require breaking down the content of the batch file into individual messages that can be directly consumed by other systems in the organization. This brings about a need for message mediation whereby a 'mediation engine' would be required to break the batch data into individually consumable messages and then transform them to the message formats that are expected by other systems in the message flow. This process can be illustrated by the diagram below.

WSO2 ESB to process batch data

WSO2 ESB is capable of acting as the mediation engine that can process the batch files to create individual messages that can be transformed to message standards, which are expected by other systems. WSO2 ESB can process batch files via the VFS (virtual file system) transport, which can listen to a given file location and pick up files for processing.

The Scenario

The scenario involves a legacy system that produces a CSV file containing customer information that needs to be processed by a CRM application. The CSV file contains information of multiple customers that needs to be processed individually by the CRM application. It is also required to transform the information in the CSV file to a message format that is expected by the CRM application. WSO2 ESB would be used to process the batch file, transform the messages, and send the messages to the CRM application. This scenario can be depicted by the diagram below.

As illustrated in the diagram above, the CSV files from the legacy system is picked up by the ESB via VFS transport. The content of the file is then transformed to the XML using a Smooks file. This file will contain all the records in the CSV file as a single XML. WSO2 ESB will then break up individual messages out of this XML file and build a message payload that would conform to the message standard expected by the CRM application. Once the message payload is built, individual messages would be sent to the CRM application. The above-mentioned process would transform a CSV file containing the following content.

0001,3463,3,12.50
0002,3452,2,4.50
0003,3463,1,6.20
0004,5421,5,3.40

Into a series of messages as given below.

0001
3463
3
12.50







0002
3452
2
4.50







0003
3463
1
6.20







0004
5421
5
3.40

Running the scenario

1. Enable VFS transport on the ESB instance - the VFS transport is disabled in the vanilla pack of the ESB. It can be enabled in the <ESB_HOME>/respository/conf/axis2/axis2.xml file. Enable the VFSTransportListener and VFSTransportSender as shown below.

2. Add the Smooks file that would transform the CSV file to XML - Include the smooks-csv.xml file in the following location <ESB_HOME>/repository/samples/resources/smooks/. The file is provided as an attachment to this article.
3. Start the WSO2 ESB
4. Add a local entry that stores the location of the Smooks file - the local entry will store the location of the Smooks file that can be used within the ESB sequence. Given below is the configuration for the local entry

5. Create a Proxy service - a proxy service needs to be created, and would need to have the mediation logic required to transform a batch file into individual messages that can be sent out to the CRM service. Given below is the proxy service configuration

      
         
         
            
               
                  
                  
   
               
               
                  
                     
                        
                           
                              
                                 $1
                                 $2
                                 $3
                                 $4
                              
                           
                           
                              
                              
                              
                              
                           
                        
                        
                        
                        
                        
                        
                           
                              
   
                           
                        
                     
                  
               
            
         
         MOVE
         5
         file:///D:/Demo/BatchFiles/processed/
         file:///D:/Demo/BatchFiles/in/
         file:///D:/Demo/BatchFiles/failure/
         .*.csv
         text/plain
         MOVE
      
   

6. Add the CSV file to the folder location provided in the vfs.FileURI path.

Summary

As demonstrated in this article, the WSO2 ESB has the capability of acting as a mediation engine that can process files containing batch data into individual messages, which can be consumed by other applications in the organization.