Integrating EDI Legacy Systems with Web Services using WSO2 ESB (Enterprise Service Bus)

Archived Content
This article is provided for historical perspective only, and may not reflect current conditions. Please refer to relevant product page for more up-to-date product information and resources.
  • By Upul G
  • 3 Mar, 2008

Applies to

WSO2 ESB 1.6

 

Table of Contents

 

Introduction

WSO2 ESB (Enterprise Service Bus) is a lightweight, robust, ultra-fast Enterprise Service Bus product released under the Apache License v2.0. It is based on the ESB of the Apache Foundation, Apache Synapse. Using WSO2 ESB you can filter, transform, route and manipulate SOAP, binary, XML, and plain text messages that pass through your business systems by HTTP, HTTPS, JMS, mail, file systems and other transport mediums.

Electronic Data Interchange (EDI) and the related standards are the means of how businesses, governments and other organizations exchange structured information among themselves. Despite the emergence of Web Services, XML and other related standards and technologies, EDI standards are still driving a considerable percentage of today's business to business communication needs. EDI has been used and still being used as a reliable, interoperable and efficient communication medium to exchange structured information among businesses and other organizations.

Businesses need to integrate their EDI based systems with today's SOAP, XML, REST based solutions. These varied systems will use different transport mediums to exchange messages among themselves such as file-based transports, Java messaging, HTTP, FTP, etc. WSO2 ESB simplifies interconnecting these disparate systems together and brings order to heterogeneous systems interactions.

 

Processing Stock Orders

In this tutorial we are going to integrate EDI source data with Web services using WSO2 ESB. WSO2 ESB will receive batches of stock orders in EDI source files. Our sample web service accepts single stock orders. The WSO2 ESB, with the help of Smooks engine, will transform EDI source files into SOAP messages, split them into separate orders and place these orders through a Web service.

 

Smooks Transform Mediator

For this use case, we are going to use the Smooks tranform mediator. This mediator uses the Milyn Smooks framework which is a versatile data transformation tool that supports data transformation between different formats such as EDI, CSV. Smooks is distributed under GNU Lesser General Public License (LGPL). In this example we are going to use its EDI to XML transformation feature.

The Smooks transform mediator extracts the ongoing message body contents, transforms it into XML and replaces the message body with the resultant output.

 

EDI Data

For this example, we will use the Simple Stock Quote Service Web service deployed in the sample server that comes bundled with the WSO2 ESB. We will be using the operation 'place order' to place new stock orders. Place order Web service operation accepts the symbol, the quantity and the price values in a request for a single stock order.

New stock order batches arrive at the ESB as EDI source files. Following is the sample EDI file we will be processing:

HDR*1200*2008-01-01
ORD*50.00*500*IBM*REF 10053
ORD*170.00*100*MSFT*--
ORD*85.00*200*SUN*A1249
ORD*138.00*100*IBM*--
ORD*18.00*500*MSFT*ref 20088398289

Listing 1 EDI Data: edi.txt

Each EDI file will contain many stock orders. It consists of a header segment followed by one or more order segments. Each order segment will contain the details of a single stock order. The order segment consists of four fields, they are stock symbol, quantity, price and remarks in that particular order.

Header Batch number Created Date

 

Order Price Quantity Symbol Remarks

Figure 1 EDI Format

ESB Configuration

Following is the ESB configuration we are going to use. We are creating a proxy service named 'PlaceStockOrderService' in the WSO2 ESB. As the name implies, a proxy service acts as a proxy running inside the WSO2 ESB. The proxy service can be exposed to the outside world through HTTP, JMS, FTP and other transports. The ESB configuration defines what happens to the messages that arrives at the proxy service. There you can transform, filter, manipulate and send to another service using different mediators.

<definitions xmlns="http://ws.apache.org/ns/synapse">
  <localEntry key="transform-xslt-key" src="file:repository/conf/sample/resources/smooks/transform.xslt"/>
  <proxy name="PlaceStockOrderService" transports="vfs">
  <parameter name="transport.vfs.FileURI">file:///home/user/test/in</parameter> <!--CHANGE-->
  <parameter name="transport.vfs.ContentType">text/plain</parameter>
  <parameter name="transport.vfs.FileNamePattern">.*\.txt</parameter>
  <parameter name="transport.PollInterval">10</parameter>
  <parameter name="transport.vfs.MoveAfterProcess">file:///home/user/test/original</parameter> <!--CHANGE-->
  <parameter name="transport.vfs.MoveAfterFailure">file:///home/user/test/original</parameter> <!--CHANGE-->
  <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
  <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
  <parameter name="Operation">urn:placeOrder</parameter>
    <target>
      <inSequence>
        <smooksTransform  smooksConfig="repository/conf/sample/resources/smooks/smooks-config.xml" />
        <xslt key="transform-xslt-key"/>
        <iterate expression="//m0:placeOrder/m0:order" preservePayload="true"
          attachPath="//m0:placeOrder" xmlns:m0="http://services.samples/xsd">
          <target>
            <sequence>
              <header name="Action" value="urn:placeOrder"/>
              <property action="set" name="OUT_ONLY" value="true"/>
              <send>
                <endpoint>
                  <address uri="http://localhost:9000/soap/SimpleStockQuoteService"/>
                </endpoint>
              </send>
            </sequence>
          </target>
        </iterate>
      </inSequence>
      <outSequence/>
      </target>
      <publishWSDL uri="file:repository/conf/sample/resources/smooks/PlaceStockOrder.wsdl"/>
    </proxy>
</definitions>

Listing 2 ESB Configuration: synapse_sample_smooks.xml

By giving transports="vfs", the proxy service will only listen to messages coming from the VFS transport listener. VFS transport listener requires some parameters to configure itself on how and where it will listen in for the messages. We are providing these details to the proxy service, through service level parameters. With the publishWSDL, we have given a custom WSDL for our proxy service.

Proxy service has two sequences namely, inSequence and outSequence. Messages destined for this proxy service will go through the inSequence mediators. If it is a response message for a request message that's coming through the proxy Web service, the response message will go through the outSequence. In this proxy service, outSequence is empty because we are not expecting any responses for the place order requests we make to the sample Axis2 server.

The VFS transport listener will pickup files from the directory given by the parameter, transport.vfs.FileURI. It will read files with the extension txt (transport.vfs.FileNamePattern), every 10 seconds (transport.PollInterval.) We have specified that the files be plain text files (transport.vfs.ContentType) not XML or binary files. It will create a message with the plain text payload and inject them to the ESB. Also, we have instructed the ESB to move the processed file to a separate directory given by the transport.vfs.MoveAfterProcess and transport.vfs.ActionAfterProcess parameters.

Incoming messages to the proxy service will go through each mediator in the inSequence. In this case, the first mediator is the smooksTransform mediator. The smooksTransform mediator uses the Smooks configuration file given by the smooksConfig attribute, to initialize the Smooks framework. In the configuration file we have used the SmooksEDIParser with the mapping.xml file given as a parameter that contains EDI to XML mapping definitions.

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.0.xsd">  
  <resource-config selector="org.xml.sax.driver">
    <resource>org.milyn.smooks.edi.SmooksEDIParser</resource>
    <param name="mapping-model">repository/conf/sample/resources/smooks/mapping.xml</param>
  </resource-config>
</smooks-resource-list>

Listing 3 Smooks Configuration File: smooks-config.xml

In the mapping.xml we have defined the characters used for separators such as segment, component and other separators. Then we have defined each segment of the EDI source file and how each segment should be mapped to an XML element.

<medi:edimap xmlns:medi="http://www.milyn.org/schema/edi-message-mapping-1.0.xsd">
  <medi:description name="place stock orders" version="1.0" />
  <medi:delimiters segment="&#10:" field="*" component="^" sub-component="~" />
  <medi:segments xmltag="orders">
    <medi:segment segcode="HDR" xmltag="header">
      <medi:field xmltag="batch-id" />
      <medi:field xmltag="date" />
    </medi:segment>
    <medi:segment segcode="ORD" xmltag="order" maxOccurs="-1">
      <medi:field xmltag="price" />
      <medi:field xmltag="quantity" />
      <medi:field xmltag="symbol" />
      <medi:field xmltag="comment" />
    </medi:segment>
  </medi:segments>
</medi:edimap>

Listing 4 EDI to XML Mapping File: mapping.xml

Note:  Replace the colon with a semicolon in segment="&#10:"

 

The root element of the resultant XML model will be named orders. EDI source file will contain two segment types: HDR and ORD. HDR will map to header XML element. There will be one HDR segment in the source file. Two fields of the HDR segment are mapped in to two XML elements: batch-id and date in that particular order. ORD segment is mapped to an order XML element. ORD segment has 4 fields in order which will be mapped into price, quantity, symbol and comment child elements of the order element.

Smooks tranformation engine will tranform the EDI source into an XML model as specified in the mapping document. It will replace the existing message payload of EDI source with the generated XML model.

<orders>
  <header>
    <batch-id>1200</batch-id>
    <date>2008-01-01</date>
  </header>
  <order>
    <price>50.00</price>
    <quantity>500</quantity>
    <symbol>IBM</symbol>
    <comment>REF 10053</comment>
  </order>
  ...
  <order>
    <price>18.00</price>
    <quantity>500</quantity>
    <symbol>MSFT</symbol>
    <comment>ref 20088398289</comment>
  </order>
</orders>

Listing 5 Output XML After EDI to XML Mapping

We need to transform the XML model to what is required by the Web service. Next mediator in the inSequence list is the XSLT mediator. It will transform the specified element of the passing message using the given stylesheet. If a specific element is not given in the configuration it will use the first child element of the SOAP body element. In this case that is what we need. We are giving the stylesheet using a local entry given at the top of the configuration.

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
  xmlns:fn="http://www.w3.org/2005/02/xpath-functions" xmlns:m0="http://services.samples/xsd"
  exclude-result-prefixes="m0 fn">
  <xsl:output method="xml" omit-xml-declaration="yes" indent="yes"/>
  <xsl:template match="/">
    <m0:placeOrder>
      <xsl:apply-templates select="//order" /> 
    </m0:placeOrder>
  </xsl:template>
  <xsl:template match="order">
    <m0:order>
      <m0:price><xsl:value-of select="price"/></m0:price>
      <m0:quantity><xsl:value-of select="quantity"/></m0:quantity>
      <m0:symbol><xsl:value-of select="symbol"/></m0:symbol>
    </m0:order>
  </xsl:template>
</xsl:stylesheet>

Listing 6 Style Sheet Used to Transform XML Model to Required Format of the Web Service: transform.xslt

The transformed XML payload of the SOAP message will be as follows. It will have many child order elements under the parent placeOrder element.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
  xmlns:xsd="http://services.samples/xsd">
  <soapenv:Header/>
  <soapenv:Body>
    <xsd:placeOrder>
      <xsd:order>
        <xsd:price>50.00</xsd:price>
        <xsd:quantity>500</xsd:quantity>
        <xsd:symbol>IBM</xsd:symbol>
      </xsd:order>
      ...
      <xsd:order>
        <xsd:price>18.00</xsd:price>
        <xsd:quantity>500</xsd:quantity>
        <xsd:symbol>MSFT</xsd:symbol>
      </xsd:order>
    </xsd:placeOrder>
  </soapenv:Body>
</soapenv:Envelope>

Listing 7 Output XML after Transformation

We need to split this multiple order XML SOAP request into orders with a single order element. The next mediator in the inSequence list will do that.

The Iterate mediator will extract the elements list given by the xpath expression attribute. In this case that is the placeOrder/order elements list. Each order element will become a separate message. By giving the attribute preservePayload as true, and the attachPath attribute, mediator will take the original SOAP message without the order elements and attach each extracted order element to the location given by the Xpath expression in attachPath attirbute. This will generate many SOAP messages with a placeOrder element with a single order element as shown below. These SOAP messages are now in an acceptable format for the Simple Stock Quote Service web service.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
  xmlns:xsd="http://services.samples/xsd">
  <soapenv:Header/>
  <soapenv:Body>
    <xsd:placeOrder>
      <xsd:order>
        <xsd:price>50.00</xsd:price>
        <xsd:quantity>500</xsd:quantity>
        <xsd:symbol>IBM</xsd:symbol>
      </xsd:order>
    </xsd:placeOrder>
  </soapenv:Body>
</soapenv:Envelope>

Listing 8 Message after Spliting the Original Message into Single Orders

Iterate mediator's target element defines what to do with the split messages. Here we have given a sequence. Each split message carrying a single order will go through this sequence. Also note, that the iterate mediator processes these split messages in parallel. We have to set the Action header to the property 'urn:placeOrder', to make the sure, the target server will dispatch the message to the place order operation of the Simple Stock Quote Web service. Place order operation is an in-only message. That means we are not expecting a response message to the sent message. By setting OUT_ONLY property to true, we tell the ESB not to expect a response message. Then we are sending the single order web service request to the sample Axis2 server.

In this case WSO2 ESB reads the EDI source files from a specified directory. Assume we want to pick up files from a remote location accessed by FTP. Instead of a local directory, you can easily change this solution so that the WSO2 ESB will listen to a remote directory accessed through FTP like this.

<parameter
name="transport.vfs.FileURI">vfs:ftp://username:password@ftp-remote-location/directory?vfs.passive=true</parameter>

 

Running the Sample

  1. Download the latest WSO2 ESB distribution package from the WSO2 ESB home page.
  2. Setup the WSO2 ESB distribution. For further details, see the Installation Guide.
  3. Deploy the SimpleStockQuoteService to sample server of the WSO2 ESB distribution. Start the sample server. For further instructions, see WSO2 ESB Samples.
  4. Download the Smooks Transform mediator from the new ESB community site, http://esbsite.org/.[Editorial comment: esbsite.org no longer exists] Copy the smooks-transform jar file to the webapp/WEB-INF/lib directory. All the files listed in these steps are available in the package at the end of this article.
  5. Download the Smooks distribution from the Smooks site. Copy the following jar files to the webapp/WEB-INF/lib directory: commons-lang-2.1.jar, milyn-commons-1.0-SNAPSHOT.jar, milyn-edisax-0.2.1.jar, milyn-smooks-core-1.0-SNAPSHOT.jar, milyn-smooks-edi-1.0-SNAPSHOT.jar.
  6. Go to repository/conf/sample/resources/ directory of the distribution. Make a new directory named smooks. We are going to keep the new artifacts here.
  7. Copy smooks-config.xml, mapping.xml, transform.xslt, PlaceStockOrder.wsdl files from the downloaded sample source package to the smooks directory. smooks-config.xml has the smooks configuration which is referred by the ESB configuration. mapping.xml has EDI to XML mapping details referred by the smooks-config.xml. transform.xslt is used by the XSLT mediator. PlaceStockOrder.wsdl is the custom WSDL for the proxy service. Note that when giving directory locations in the XML files, they are given relative to the ESB home directory.
  8. Open a web browser and go to the administration console by going to https://localhost:9443/esb
  9. Log in using the default username and password as admin and admin.
  10. Go to the Configuration page and replace the existing configuration with the above ESB configuration in Listing 2. This is also available in the sample source package: synapse_sample_smooks.xml.
  11. You have to change the transport.vfs.FileURI to a valid directory where your VFS listener will pick files from. Create a new empty directory and give the location with transport.vfs.FileURI. Also create two directories for processed and failed files and give them through transport.vfs.MoveAfterProcess and transport.vfs.MoveAfterFailure respectively.
  12. Click Update. You need to click on Save button to save the configuration permanently. Otherwise configuration changes will be lost when you restart the WSO2 ESB instance.
  13. Now put the edi.txt file in Listing 1 (also available in the sample source package) into the directory location given by the transport.vfs.FileURI.

WSO2 ESB will process the file, split it into individual orders, send the orders and move the original file to the processed directory.

You will see the following logs in the sample Axis2 server console. Stock order details that were in the EDI file has been sent to the Web service:

Thu Feb 07 14:25:25 IST 2008 samples.services.SimpleStockQuoteService  :: Accepted order for : 500 stocks of IBM at $ 50.0
Thu Feb 07 14:25:25 IST 2008 samples.services.SimpleStockQuoteService  :: Accepted order for : 500 stocks of MSFT at $ 18.0
Thu Feb 07 14:25:25 IST 2008 samples.services.SimpleStockQuoteService  :: Accepted order for : 100 stocks of IBM at $ 138.0
Thu Feb 07 14:25:25 IST 2008 samples.services.SimpleStockQuoteService  :: Accepted order for : 100 stocks of MSFT at $ 170.0
Thu Feb 07 14:25:25 IST 2008 samples.services.SimpleStockQuoteService  :: Accepted order for : 200 stocks of SUN at $ 85.0

Figure 2 Sample Server Console Output

This way you can successfully integrate EDI data with Web services using WSO2 ESB.

 

Author

Upul Godage, Senior Software Engineer, WSO2 Inc. upul AT wso2 DOT com

About Author

  • Upul G
  • IT