WSO2 ESB Performance Testing Round 1

Performance Environment

In order to test the scenarios, we used three separate systems. They were for load generation, ESB deployment and to host the service implementation backend. The hardware and software configurations and the OS level tuning parameters used are listed under the 'Notes' section below. Please note: in order to get high levels of concurrent connections it is essential to tune the TCP/IP stack.

Load Generator

We used a Java clone of the popular ApacheBench load generator. This code, from the Apache HttpCore project, can properly support HTTP 1.1 (chunked) output and also behaves well with HTTPS. For Web services invocations, it is almost as fast as its native counterpart with only a negligible difference in the visible performance. You may download and build this tool from the HttpCore project source [1] or use it from a HttpCore release. Alternatively, I have made the client available in compiled form [1]. You can download the benchmark.jar and commons-cli-1.0.jar into a working directory, and issue "java -jar benchmark.jar" to view the list of available options. The options are the same as those found in ApacheBench.

For example, "java -jar benchmark.jar -p request.xml -n 10000 -c 20 -k -H "SOAPAction: urn:getQuote" -T "text/xml; charset=UTF-8" https://localhost:8080/soap/Service"

The Back-end Service Host

We used an Apache Axis2 1.2 SOAP engine back end, which is bundled by default with Apache Synapse [2] and the WSO2 ESB [3], and the SimpleStockQuoteService (with the System.out.println statement commented out for performance). For testing scenario 1, we used a simple simulation of a Web service through a servlet hosted on Apache Tomcat 5.5.23. This simulator was able to generate a 1K response for a 1K request and a 5K response for a 5K request with minimal overhead or delay. This is an ideal test-bed for an ESB - both the client and server are very lightweight and therefore able to provide very high levels of load - with the result that the intermediary is tested to the maximum.

Scenario 1: Virtualization / Passthrough of a 1K request / 1K response

The client kept generating 1K requests that yielded 1K responses served by the Tomcat service simulation, and the ESB was used to virtualize the actual service endpoint. The client requests were directed to a virtual service endpoint created on the ESB, that simply directed the requests to the service implementation backend, and returned the response back through the ESB to the client. The number of concurrent clients were doubled starting from 20 users to 2560 concurrent users. The aim of this test is to get a baseline for performance by testing the simplest possible intermediary case.

The results show the scalability of a single instance of the WSO2 ESB (and of course the underlying Apache Synapse engine). The results show that the ESB can handle a load very effectively and that it is capable of handling thousands of concurrent/simultaneous connections.

Scenario 2: Content based routing based on an XPath expression on the payload

The client generated a stock quote request of approximately 310 bytes, that yielded a response of approximately 850 bytes. The backend service was an Axis2 1.2 based Simple StockQuoteService that ships with Apache Synapse and the WSO2 ESB. It was hosted on the sample Axis2 server bundled with both distributions.

The ESB was used to make a routing decision based on the requested stock symbol for a sample SOAP payload as follows.

<m:getQuote xmlns:m="https://services.samples/xsd">
  <m:request>
    <m:symbol>IBM</m:symbol>
  </m:request>
</m:getQuote>

We compared the performance of the WSO2 ESB against one of the leading proprietary (closed-source) Java ESB's for the same scenario. The proprietary ESB was run in its production mode, while the client/service backend and the OS level tuning was left untouched. The WSO2 ESB used around 307K of RAM and 86.25% of the CPU on average during the tests, while the proprietary ESB utilized 733K of RAM and 67.75% of the CPU on average. The WSO2 ESB was able to surpass the proprietary ESB in performance for all levels of concurrency tested.

Scenario 3: XSLT transformation of the request and the response

The client generated a custom stock quote of approximately 310 bytes that was transformed into a standard stockquote request by the ESB of approximately the same size, through an XSLT transformation. The backend service response of around 850 bytes were then transformed back into a custom format as expected by the end client through another transformation. This is similar to sample # 8 of Apache Synapse / WSO2 ESB.

<m0:checkPriceRequest xmlns:m0="https://www.apache-synapse.org/test">
  <m0:Code>symbol</m0:Code>
</m0:checkPriceRequest>

We compared the performance of the WSO2 ESB against the same proprietary ESB under equivalent conditions. The WSO2 ESB used around 331K of RAM and 92.75% of the CPU during the tests on average, while the proprietary ESB utilized 745K of RAM and 77.25% of the CPU on average. The WSO2 ESB was able to perform almost twice the number of XSLT transformations per second with a lesser response time than the proprietary ESB.

Conclusion

The results show the initial performance benchmark results of the WSO2 ESB / Apache Synapse, and proves its scalability to handle thousands of concurrent connections. It also shows that the WSO2 ESB is able to perform more XPath based CBR routing decisions, as well as XSLT transformations than one of the leading proprietary ESBs, with the results being almost twice as better in some scenarios. In addition, the memory footprint of the WSO2 ESB is about half the size of the competitor, meaning that - for example - the WSO2 ESB could be co-located with an existing service client or server.

Notes

* All performance numbers were captured while running a single instance of the ESB on a single JVM without any special JVM arguments.

Client / Load generator (testa) and the Back-end service host (testb)

Intel(R) Xeon(TM) 3.20GHz 2MB Cache Dual Core - 2 CPU system / 2GB RAM / 1Gbps Ethernet

Fedora Core release 5 (Bordeaux) - 2.6.15-1.2054_FC5smp #1 SMP

Client - Sun JDK 1.5.0_06-b05 / Load generator - Sun JDK 1.5.0_10-b03

/etc/sysctl.conf net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
/etc/security/limits.conf
* soft nofile 4096
* hard nofile 65535

WSO2 ESB

Intel(R) Xeon(R) 2.00GHz Dual core - 2 CPU system / 2 GB RAM / 1Gbps Ethernet

Red Hat Enterprise Linux ES release 4 (Nahant) - 2.6.9-5.ELsmp #1 SMP / Sun JDK 1.5.0_10-b03

/etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_default = 524288
net.core.wmem_default = 524288
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
/etc/security/limits.conf
* soft nofile 4096
* hard nofile 65535

Resources

[1] Java clone of Apache Bench from Jakarta HttpCore or the Binary

[2] Apache Synapse 1.0 download

[3] WSO2 ESB download

References

WSO2 ESB Performance Testing Part 2

Author

Asankha Perera, Architect, WSO2 Inc., asankha at wso2 dot com