This article presents the latest (as of January 2013) performance study conducted to compare the performance of WSO2 ESB 4.6.0, WSO2 ESB 4.5.1, Mule 3.3.0, Talend-SE-5.1.1, and UltraESB 1.7.1 - Enhanced. WSO2 has been performing and publishing ESB performance data since 2007, and this paper continues our ongoing work to ensure that we evaluate and publish useful performance numbers for our customers. In addition, the WSO2 ESB 4.6.0 incorporates significant performance updates to ensure that it provides class-leading performance in a number of scenarios.
WSO2 ESB 4.6.0 is the latest version of the ESB at the time of writing, and the data below shows several performance improvements over ESB 4.5.x. The performance benchmarks are performed in Amazon EC2 as done in the ESB performance Round 6, so that they can be independently verified and repeated.
We have chosen to called this test “Round 6.5” because we are comparing against the same versions and results of the ESBs used in the Round 6 performance benchmark published on the UltraESB-managed performance site esbperformance.org. For the sake of comparison, we have included the Round 6 results for UltraESB 1.7.1, Mule ESB 3.3.0, and Talend-SE-5.1.1.
WSO2 ESB uses the Passthrough Transport (PTT) as the default transport. The PTT has improved performance over the Non-Blocking HTTP Transport (NHTTP), which was the previous default transport available with WSO2 ESB. Both transports utilize the Apache HTTPCore NIO project to provide high-performance, low-latency support for HTTP.
The main difference between PTT and NHTTP is that when the PTT is the default and both the incoming transport and outgoing transport are HTTP, the PTT can utilize a single shared buffer for relaying network data from input network connections to output connections. In similar circumstances, the NHTTP transport utilizes two buffers. This improvement is particularly useful for the performance of pass-through mediation. You can find more details on performance in this article.
We have improved the mediation engine to be content aware, so it can differentiate between use cases that do read and do not read the message being mediated. Consequently, it can intelligently identify scenarios where the engine can safely stream data from the incoming connection to outgoing connections without reading them to memory. With earlier WSO2 releases, users had to identify and engage pass-through support as needed.
WSO2 ESB 4.6.0 particularly focuses on XPath support for content-based routing. The new XPath support includes a partial compilation strategy to target specific scenarios. The following article goes into much more detail about these improvements:
A significantly improved XSLT capability called “Fast XSLT” provides much faster streaming transformation when used with the PTT. The new streaming XSLT mediation capability provides significant performance gains in many cases (especially when messages that can be fit into the default buffer performed faster than previous XSLT mediator) and also results in significantly better memory usage. See the EnhancedXSLTProxy results in the graphs below to evaluate the differences.
Following are the tuning parameters we used with WSO2 ESB 4.6.0. For the other ESBs, we used the same configurations as in the Round 6 performance benchmark.
The memory allocated for the ESB was increased by modifying
Default setting for WSO2 ESB 4.6.0: -Xms256m -Xmx512m -XX:MaxPermSize=256m
New setting used for benchmarking: -Xms2048m -Xmx2048m -XX:MaxPermSize=1024m
Disable access logs in
OS level optimizations - in /etc/sysctl.conf:
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
And in /etc/security/limits.conf:
* soft nofile 4096
* hard nofile 65535
All the load tests were carried out using Amazon AWS EC2 with the instance type "High-CPU Extra Large (c1.xlarge, 7GB).” The configuration used is described below. The back-end echo service used for testing was hosted in fully optimized Apache Tomcat 7.0.29.
|model name||Intel(R) Xeon(R) CPU E5410 @ 2.33GHz|
|Nos of Cores||8|
|RAM Size||7GB (High-CPU Extra Large)|
Server : apache-tomcat-7.0.2
Client : apache-ab
The test setup includes a back-end service, clients, and ESB sitting in between and performing the following mediation scenarios. The back-end service is a minimal implementation that introduces minimal overhead.
For more details on our testing approach and setup, see WSO2 Enterprise Service Bus (ESB) Performance Testing - Round 3
For each scenario, we tested using different concurrency values and different message sizes. Sample message sizes ranged from 512 bytes to 100K bytes, with concurrency levels of 20, 40, 80, 160, 320, 640, 1280, and 2560 users.
The following table and graph show the results of the performance test. Note that this is similar to the graph present in the earlier Round 6 article except that this uses a bar graph instead of the line graph, because the X axis is continuous. This graph takes the average across all message sizes.
First we ran WSO2 ESB 4.6.0 using the same setup and configurations explained in round 6, and the following table and graph shows the results.
|Mule 3.3.0||Talend-SE-5.1.1||UltraESB 1.7.1 -Enhanced||WSO2 ESB 4.5.1||WSO2 ESB 4.6.0|
(Fast XSLT mediator)
However, we observed that the data in this test was perhaps unreliable. We investigated this in detail and observed that in the published test, the high concurrency results were showing unexpected high values. For the Round 6 tests, there was a very low number of messages per client (n=10) for high concurrency. This does not allow the JVM time to warm up and also does not represent a long running test.
Therefore, we chose to re-run the test using 200 messages per client with higher concurrencies, and then the anomalous data was no longer observed. We also re-ran the UltraESB in an EC2 image with 200 messages from each client for the higher concurrencies following the instructions given in the Round 6 article. Our observations for n=200 and high concurrency with UltraESB were much lower than those previously published. We gave the UltraESB team the benefit of the doubt, and we have published our lower numbers (with n=200 for high concurrency) against UltraESB’s existing published numbers (with n=10 for high concurrency). We also publish the existing numbers for Mule and Talend. We did not rerun the Mule or Talend tests using n=200. The results are shown in the following graph and table.
We would encourage future test runs to utilize n=200 for higher concurrencies to ensure reasonable results.
|Mule 3.3.0||Talend-SE-5.1.1||UltraESB 1.7.1-Enhanced||WSO2 ESB 4.5.1||WSO2 ESB 4.6.0|
(Fast XSLT mediator)
The above graph demonstrates that WSO2 ESB 4.6.0 is consistently faster than ESB 4.5.1 and performs competitively with UltraESB in the scenarios. While the XSLTProxy remains significantly less performant than the equivalent UltraESB test, this is addressed by the inclusion in WSO2 ESB 4.6.0 of the new FastXSLT option, which provides performance on par with UltraESB.
To measure the latency added by WSO2 ESB versions 4.6.0 and 4.5.1, we calculated the latency for concurrency of 20 and message size 1000 using the following formula: [Latency = Direct Backend Call - Round Trip via ESB]. The following table lists the results:
|Latency for 1K message||ESB 4.6.0||ESB 4.5.1|
As indicated in the table and graph, WSO2 ESB provides sub-millisecond latency overhead in most scenarios.
For complete analysis, you can view the Google spreadsheet of collected data.
The following graphs illustrate the memory usage after 40 hours. As you can see, heap memory utilization is stable.
This article presents a performance study comparing WSO2 ESB 4.6.0, WSO2 ESB 4.5.1, Mule 3.3.0, Talend-SE 5.1.1, and UltraESB 1.7.1. The results indicate that WSO2 ESB 4.6.0 has significantly improved performance over previous versions and performs better than the other compared ESBs in most scenarios.
WSO2 and WSO2 ESB are trademarks of WSO2 Inc. UltraESB and AdroitLogic are trademarks of AdroitLogic Private Ltd. Mule ESB and MuleSoft are trademarks of MuleSoft, Inc. Talend and Talend ESB are trademarks of Talend, Inc. All other product and company names and marks mentioned are the property of their respective owners and are mentioned for identification purposes only.