×
2015/05/29
29 May, 2015

[Tutorial] Using The File System as a Transport Medium with WSO2 ESB VFS Transport

  • Suhan Dharmasuriya
  • Associate Technical Lead - WSO2

Table of Contents

  1. Introduction
  2. Configuring WSO2 ESB
  3. Analysing the output
  4. Improvements
    • Saving VFS transport logs in a separate log file
    • Streaming large files
    • Storing the endpoint in governance registry
  5. Future Improvements
    • Fetching FileURI service parameter from registry
  6. Summary
  7. References

Introduction

This step-by-step guide shows you how the WSO2 Enterprise Service Bus (ESB) can be easily configured to use its file system as a transport medium using VFS Transport. The request is stored in an xml file. The ESB will process this file and send the request to a backend axis2 service. The response received will also be saved in an xml file. We will save the VFS transport logs in a separate log file.

Figure 01

By using the sample SimpleStockQuoteService service shipped within WSO2 ESB, we are going to test this scenario. We will send a SOAP request to the SimpleStockQuoteService and get back the response from it. Here both the request and the response will be saved in separate xml files.

I'm going to create a directory structure as follows:

Suhans-MacBook-Pro:test suhanr$ tree

├── in

├── original

└── out

i.e.

  • /Users/suhanr/Desktop/test/in - the location to put SOAP request xml files to be processed
  • /Users/suhanr/Desktop/test/original - location to put processed xml files (which were in ../in folder before processing)
  • /Users/suhanr/Desktop/test/out - Location to store the responses received from service as xml files

Our proxy will automatically poll for files in the given location (i.e. ../in location above). The transport.PollInterval is 15 seconds in this scenario (the default polling interval).

If a file is found in ../in location, it will be processed and will be moved to ../original folder. Then the response is saved in a file in ../out folder.

Configuring WSO2 ESB

  1. You can download WSO2 ESB 4.8.1 from here. If you already have the product zip file (wso2esb-4.8.1.zip) continue with the next step.
  2. Unzip the product to a path containing no spaces in the path name. This is your <ESB_HOME>

    e.g. /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1

  3. Prepare the SimpleStockQuoteService service:
    • Go to ESB_HOME/samples/axis2Server/src/SimpleStockQuoteService. You can see the files and folders as follows:

      build.xml conf src

    • Issue an ant command to build the SimpleStockQuoteService.aar
      Suhans-MacBook-Pro:SimpleStockQuoteService suhanr$ ant
Buildfile: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/build.xml

clean:

init:
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/classes
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services
compile-all:
    [javac] /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/build.xml:47: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 9 source files to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/classes
build-service:
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote/META-INF
     [copy] Copying 1 file to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote/META-INF
     [copy] Copying 9 files to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote
      [jar] Building jar: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services/SimpleStockQuoteService.aar
BUILD SUCCESSFUL
Total time: 1 second

      • Now your SimpleStockQuoteService.aar jar is copied to services at ESB_HOME/samples/axis2Server/repository/services/.

  4. Configure VFS Transport in axis2.xml file:
    • Open another terminal, console tab or window and go to ESB_HOME/repository/conf/axis2
    • Open axis2.xml file to edit

      Suhans-MacBook-Pro:wso2esb-4.8.1 suhanr$ vi repository/conf/axis2/axis2.xml

    • Uncomment the VFS listener and VFS sender as follows:
      <transportreceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>
...
<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>
    • Save the changes
  5. Create the SOAP request input xml file:
    • Create the folder structure shown in the overview section at the beginning of this article. You can rename these folders as you like and create them anywhere that is accessible to the server.

      i.e. three folders namely, in, original and out.

    • Go to ../in folder that you have created and create a file named test.xml and copy the following content to it.
      <?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="https://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="https://www.w3.org/2005/08/addressing">
    <soapenv:Body>
        <m0:getQuote xmlns:m0="https://services.samples">
            <m0:request>
                <m0:symbol>WSO2</m0:symbol>
            </m0:request>
        </m0:getQuote>
    </soapenv:Body>
</soapenv:Envelope>
  6. Start axis2 server
    • Go to ESB_HOME/samples/axis2Server
    • Start server by issuing the sh axis2server.sh command
      Suhans-MacBook-Pro:axis2Server suhanr$ sh axis2server.sh 
Using JAVA_HOME:   /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home
Using AXIS2 Repository :   /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository
Using AXIS2 Configuration :   /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/conf/axis2.xml
15/03/20 10:41:31 INFO util.SampleAxis2ServerManager: [SimpleAxisServer] Starting
[SimpleAxisServer] Using the Axis2 Repository : /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository
[SimpleAxisServer] Using the Axis2 Configuration File : /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/conf/axis2.xml
15/03/20 10:41:31 INFO deployment.ModuleDeployer: Deploying module: addressing - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/addressing.mar
15/03/20 10:41:31 INFO deployment.ModuleDeployer: Deploying module: rampart - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/rampart.mar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: sandesha2 - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/sandesha2.mar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: addressing - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.addressing_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2caching - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.caching.core_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: ComponentMgtModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.feature.mgt.services_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2mex - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.mex_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: pagination - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.registry.server_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: relay - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.relay.module_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: sandesha2 - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.rm_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: POXSecurityModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.security.mgt_4.2.2.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: ServerAdminModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.server.admin_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2statistics - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.statistics_4.2.2.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2throttle - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.throttle.core_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: usagethrottling - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.throttling.agent_2.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2tracer - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.tracer_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: metering - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.usage.agent_2.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2xfer - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.xfer_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: rampart - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/rampart-core_1.6.1.wso2v12.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: rahas - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/rampart-trust_1.6.1.wso2v12.jar
15/03/20 10:41:32 ERROR sandesha2.SandeshaModule: Could not load module policies. Using default values.
15/03/20 10:41:32 INFO config.ClientConnFactoryBuilder: HTTPS Loading Identity Keystore from : ../../repository/resources/security/wso2carbon.jks
15/03/20 10:41:32 INFO config.ClientConnFactoryBuilder: HTTPS Loading Trust Keystore from : ../../repository/resources/security/client-truststore.jks
15/03/20 10:41:32 INFO nhttp.HttpCoreNIOSender: HTTPS Sender starting
15/03/20 10:41:32 INFO nhttp.HttpCoreNIOSender: HTTP Sender starting
15/03/20 10:41:32 INFO jms.JMSSender: JMS Sender started
15/03/20 10:41:32 INFO jms.JMSSender: JMS Transport Sender initialized...
15/03/20 10:41:32 INFO deployment.DeploymentEngine: Deploying Web service: SimpleStockQuoteService.aar - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services/SimpleStockQuoteService.aar
15/03/20 10:41:33 INFO nhttp.HttpCoreNIOListener: HTTPS Listener started on 0:0:0:0:0:0:0:0:9002
15/03/20 10:41:33 INFO nhttp.HttpCoreNIOListener: HTTP Listener started on 0:0:0:0:0:0:0:0:9000
15/03/20 10:41:33 INFO util.SampleAxis2ServerManager: [SimpleAxisServer] Started
    • To make sure that the service is deployed, go to https://localhost:9000/services/SimpleStockQuoteService?wsdl and check whether the WSDL file can be viewed
  7. Start the WSO2 ESB server

    To start the server, your have to run the script wso2server.bat (on Windows) or wso2server.sh (on Linux/Solaris) from the <ESB_HOME>/bin folder

  8. Log in to the ESB using default credentials (username: admin, password: admin)
  9. Add a proxy StockQuoteProxy from source view or create a custom proxy configuration in ESB
    <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true">
      <target>
         <endpoint>
            <address uri="https://localhost:9000/services/SimpleStockQuoteService" format="soap12"/>
         </endpoint>
         <outSequence>
            <property name="transport.vfs.ReplyFileName" expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"  scope="transport"/>
            <property name="OUT_ONLY" value="true"/>
            <send>
               <endpoint>
                  <address uri="vfs:file:///Users/suhanr/Desktop/test/out"/>
               </endpoint>
            </send>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/suhanr/Desktop/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
      <parameter name="transport.vfs.ContentType">text/xml</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
  10. After adding the StockQuoteProxy proxy configuration, your ESB synapse configuration1 will look like the code shown below:
    <?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
      <parameter name="cachableDuration">15000</parameter>
   </registry>
   <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true">
      <target>
         <endpoint>
            <address uri="https://localhost:9000/services/SimpleStockQuoteService" format="soap12"/>
         </endpoint>
         <outSequence>
            <property name="transport.vfs.ReplyFileName" expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')" scope="transport"/
            <property name="OUT_ONLY" value="true"/>
            <send>
               <endpoint>
                  <address uri="vfs:file:///Users/suhanr/Desktop/test/out"/>
               </endpoint>
            </send>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/suhanr/Desktop/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
      <parameter name="transport.vfs.ContentType">text/xml</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>
   <sequence name="fault">
      <log level="full">
         <property name="MESSAGE" value="Executing default 'fault' sequence"/>
         <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
         <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
      </log>
      <drop/>
   </sequence>
   <sequence name="main">
      <in>
         <log level="full"/>
         <filter source="get-property('To')" regex="https://localhost:9000.*">
            <send/>
         </filter>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>
</definitions>

    Once you save the configuration, your file will be processed within 15 seconds. Now let’s verify the results.

Analysing the output

Once you save your configuration, the ESB console output will be as follows:

[2015-03-20 11:32:53,581]  INFO - SynapseTaskManager Shutting down the task manager
[2015-03-20 11:32:53,583]  INFO - XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
[2015-03-20 11:32:53,589]  INFO - ProxyService Building Axis service for Proxy service : StockQuoteProxy
[2015-03-20 11:32:53,610]  INFO - ProxyService Adding service StockQuoteProxy to the Axis2 configuration
[2015-03-20 11:32:53,622]  INFO - DeploymentInterceptor Deploying Axis2 service: StockQuoteProxy {super-tenant}
[2015-03-20 11:32:53,695]  INFO - ProxyService Successfully created the Axis2 service for Proxy service : StockQuoteProxy

When the file is processed, you can see this output in your Axis2 server console:

Fri Mar 20 11:33:08 IST 2015 samples.services.SimpleStockQuoteService :: Generating quote for : WSO2

Your folder tree structure will look as follows:

Suhans-MacBook-Pro:test suhanr$ tree

├── in

├── original

│ ├── test.xml

└── out

└── a9b795da-a668-4b6a-ac8a-90db42d7311d.xml

As you can see the test.xml file is processed and moved to the ../original folder.

The response received from the StockQuoteProxy is placed in ../out folder as a9b795da-a668-4b6a-ac8a-90db42d7311d.xml

The content of the response file is as follows:

<soapenv:envelope xmlns:soapenv="https://www.w3.org/2003/05/soap-envelope">
   <soapenv:body>
      <ns:getquoteresponse xmlns:ns="https://services.samples">
         <ns:return xmlns:ax21="https://services.samples/xsd" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:type="ax21:GetQuoteResponse">
            <ax21:change>-2.378108963314817</ax21:change>
            <ax21:earnings>12.594090499041867</ax21:earnings>
            <ax21:high>-79.46865111676226</ax21:high>
            <ax21:last>79.76869958422132</ax21:last>
            <ax21:lasttradetimestamp>Fri Mar 20 11:33:08 IST 2015</ax21:lasttradetimestamp>
            <ax21:low>-79.2594140983345</ax21:low>
            <ax21:marketcap>-7611066.458300805</ax21:marketcap>
            <ax21:name>WSO2 Company</ax21:name>
            <ax21:open>81.77710215829231</ax21:open>
            <ax21:peratio>24.900435994896995</ax21:peratio>
            <ax21:percentagechange>3.054782179111802</ax21:percentagechange>
            <ax21:prevclose>-77.84872452039339</ax21:prevclose>
            <ax21:symbol>WSO2</ax21:symbol>
            <ax21:volume>7308</ax21:volume>
         </ns:return>
      </ns:getquoteresponse>
   </soapenv:body>
</soapenv:envelope>

Improvements

Saving VFS transport logs in a separate log file

  1. Append the following configuration to <ESB-HOME>/repository/conf/log4j.properies file2 
    # VFS separate config log file
log4j.logger.org.apache.synapse.transport.vfs=DEBUG, CARBON_VFS_LOGFILE
log4j.additivity.org.apache.synapse.transport.vfs = false
log4j.appender.CARBON_VFS_LOGFILE=org.wso2.carbon.logging.appenders.CarbonDailyRollingFileAppender
log4j.appender.CARBON_VFS_LOGFILE.File=${carbon.home}/repository/logs/${instance.log}/wso2carbon-vfs${instance.log}.log
log4j.appender.CARBON_VFS_LOGFILE.Append=true
log4j.appender.CARBON_VFS_LOGFILE.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout
log4j.appender.CARBON_VFS_LOGFILE.layout.ConversionPattern=TID: [%T] [%S] [%d] %P%5p {%c} - %x %m {%c}%n
log4j.appender.CARBON_VFS_LOGFILE.layout.TenantPattern=%U%@%D [%T] [%S]
log4j.appender.CARBON_VFS_LOGFILE.threshold=DEBUG

    New wso2carbon-vfs.log file will be created in <ESB-HOME&gt/repository/logs directory which contains all VFS transport DEBUG logs. These logs will NOT be included in wso2carbon.log file.

  2. Restart the ESB server
  3. Add another xml file (test2.xml) into the ../in directory to process, with a different symbol name such as IBM.
  4. Within 15 seconds your file will be processed (see the sample axis2 server console)
    Wed May 27 18:13:04 IST 2015 samples.services.SimpleStockQuoteService :: Generating quote for : IBM
  5. You can view the wso2carbon-vfs.log file as follows:
    Suhans-MacBook-Pro:files suhanr$ tail -f ../repository/logs/wso2carbon-vfs.log 
TID: [0] [ESB] [2015-05-27 18:12:33,868] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Scanning directory or file : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:12:33,869] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  File name pattern : .*\.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:12:33,869] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Non-Matching file : .DS_Store {org.apache.synapse.transport.vfs.VFSTransportListener}

TID: [0] [ESB] [2015-05-27 18:13:03,875] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Scanning directory or file : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:03,878] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  File name pattern : .*\.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:03,879] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Non-Matching file : .DS_Store {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:03,879] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Matching file : test2.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:03,886] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Processing file :file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in/test2.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:04,501] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Processed file : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in/test2.xml of Content-type : text/xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:04,501] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Moving to file :file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/original/test2.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:04,560] DEBUG {org.apache.synapse.transport.vfs.VFSOutTransportInfo} -  Using the fileURI        : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/out {org.apache.synapse.transport.vfs.VFSOutTransportInfo}
TID: [0] [ESB] [2015-05-27 18:13:04,560] DEBUG {org.apache.synapse.transport.vfs.VFSOutTransportInfo} -  Using the maxRetryCount  : 3 {org.apache.synapse.transport.vfs.VFSOutTransportInfo}
TID: [0] [ESB] [2015-05-27 18:13:04,560] DEBUG {org.apache.synapse.transport.vfs.VFSOutTransportInfo} -  Using the reconnectionTimeout : 30000 {org.apache.synapse.transport.vfs.VFSOutTransportInfo}
TID: [0] [ESB] [2015-05-27 18:13:04,560] DEBUG {org.apache.synapse.transport.vfs.VFSOutTransportInfo} -  Using the append         : false {org.apache.synapse.transport.vfs.VFSOutTransportInfo}
TID: [0] [ESB] [2015-05-27 18:13:04,560] DEBUG {org.apache.synapse.transport.vfs.VFSOutTransportInfo} -  File locking             : ON {org.apache.synapse.transport.vfs.VFSOutTransportInfo}
TID: [0] [ESB] [2015-05-27 18:13:19,505] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Scanning directory or file : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:19,506] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  File name pattern : .*\.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:19,506] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Non-Matching file : .DS_Store {org.apache.synapse.transport.vfs.VFSTransportListener}

TID: [0] [ESB] [2015-05-27 18:13:34,512] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Scanning directory or file : file:///Users/suhanr/Documents/compressed/wso2esb-4.8.1/files/in {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:34,513] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  File name pattern : .*\.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2015-05-27 18:13:34,513] DEBUG {org.apache.synapse.transport.vfs.VFSTransportListener} -  Non-Matching file : .DS_Store {org.apache.synapse.transport.vfs.VFSTransportListener}

Streaming large files

  1. Add the following as a VFS property (e.g. next to transport.vfs.ReplyFileName property).

    <property name="transport.vfs.Streaming" value="true" scope="transport" type="STRING"/>

  2. Set transport.vfs.Append=true parameter with the file address URI in order to append to the output file4.

    <address uri="vfs:file:///Users/suhanr/Desktop/test/out?transport.vfs.Append=true"/>

Storing the endpoint in governance registry

You can store the endpoint in governance registry or configuration registry4.

e.g. Governance registry endpoint: "StockQuoteEP"

<endpoint xmlns="http://ws.apache.org/ns/synapse">
   <address uri="https://localhost:9000/services/SimpleStockQuoteService" format="soap12">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>

The modified proxy service will be as follows. Note the endpoint key value gov:/StockQuoteEP.

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
      <parameter name="cachableDuration">15000</parameter>
   </registry>
   <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true">
      <target>
         <endpoint key="gov:/StockQuoteEP"/>
         <outSequence>
            <property name="transport.vfs.ReplyFileName" expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')" scope="transport"/>
            <property name="OUT_ONLY" value="true"/>
            <send>
               <endpoint>
                  <address uri="vfs:file:///Users/suhanr/Desktop/test/out"/>
               </endpoint>
            </send>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/suhanr/Desktop/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
      <parameter name="transport.vfs.ContentType">text/xml</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>
   <sequence name="fault">
      <log level="full">
         <property name="MESSAGE" value="Executing default 'fault' sequence"/>
         <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
         <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
      </log>
      <drop/>
   </sequence>
   <sequence name="main">
      <in>
         <log level="full"/>
         <filter source="get-property('To')" regex="https://localhost:9000.*">
            <send/>
         </filter>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>
</definitions>

Future Improvements

Fetching FileURI service parameter from the registry

WSO2 ESB 4.8.1 does not support fetching transport.vfs.FileURI from the registry (since ESB does not support fetching service parameters from registry presently).

However in the next release i.e. ESB 4.9.0 we will be able to cater to this requirement through inboundEndpoint3.

The sample configuration is given below 5 (Applicable to ESB 4.9.0 and later releases only). Please note that the inboundEndpoint configuration shown below is subjected to change since WSO2 ESB 4.9.0 was not released when this article was published.

<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse";  name="wso2File"
 sequence="request"  onError="fault"  protocol="file"
 suspend="false">
   <parameters>
      <parameter name="interval">1000</parameter>
      <parameter name="transport.vfs.ActionAfterErrors">NONE</parameter>
      <parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
      <parameter name="transport.vfs.AutoLockRelease">false</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">NONE</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">NONE</parameter>
      <parameter name="sequential">false</parameter>
      <parameter name="transport.vfs.FileURI" *key="conf:/repository/esb/esb-configurations/test"*/>
      <parameter name="transport.vfs.DistributedLock">false</parameter>
      <parameter name="transport.vfs.Locking">enable</parameter>
   </parameters>
</inboundEndpoint>

Summary

In this tutorial we have looked at using the file system as a transport medium using WSO2 ESB 4.8.1 VFS Transport. For this use case we have used the sample SimpleStockQuoteService service. SOAP request is stored in an xml file and the response is received from the SimpleStockQuoteService, which is also stored in an xml file.

WSO2 ESB is the main integration backbone of the WSO2 platform. Unlike some Enterprise Service Buses in the market, WSO2 ESB supports all enterprise integration patterns. This is one of its main competitive advantages. System administrators and SOA architects can effortlessly configure message routing, virtualization, intermediation, transformation, logging, task scheduling, load balancing, fail-over routing, event brokering, and much more with WSO2 ESB.

What if you want to go even further? Is it possible with WSO2 ESB? Yes, there are extension points to WSO2 ESB such as, script mediator, class/custom mediators, connectors, custom tasks, message builders/formatters and even allows custom transports. Most importantly WSO2 ESB can co-exist with legacy systems and allows you to fully harness their current backend capabilities.

References

[1] https://suhan-opensource.blogspot.com/2015/03/how-to-view-wso2-esb-synapse.html

[2] https://gist.github.com/R-Rajkumar/d92a139702d93998926c

[3] https://docs.wso2.com/display/ESB481/VFS+Transport#VFSTransport-parametersVFSservice-levelparameters

[4] https://suhan-opensource.blogspot.com/2015/05/wso2-esb-how-to-save-sequence-in.html

[5] https://www.mail-archive.com/[email protected]/msg44308.html

[6] https://suhan-opensource.blogspot.com/2015/03/using-file-system-as-transport-medium.html

 

About Author

  • Suhan Dharmasuriya
  • Associate Technical Lead
  • WSO2 Lanka (Pvt) Ltd.

API Management
Integration
IAM
Internal Developer Platform
Solutions
Resources
Support
Company
Sign Up For Our Newsletter

Follow us

Learn About Security At WSO2 →