Axis2/Java – Performance Testing Round #1

Test Suite

We decided to follow a set of academic work that has been done in this area. We started off with Alek and Madhu’s “SOAP Benchmark Suite v1” and “SOAP Benchmark Suite v2”. The original SOAP Bench v1 had send/receive/echo operations for arrays of bytes (BASE64 encoded), integers, doubles, and strings. SOAP Bench v2 added send/receive/echo operations for two more data structures: SimpleEvent (a complexType with a sequenceNumber/integer, timestamp/double, message/string) and MeshInterfaceObject (a complexType with 2 integers x, y and a double value). Please see details on the v1 test suite here - grid2004.pdf and the v2 test suite here - soapBench-SC05.pdf.

Deciding what to test and how

Once we decided on the benchmark suite we came across an issue: the benchmark WSDL is RPC-Encoded. WS-I BaseProfile 1.0 bans RPC-Encoded, and Axis2 doesn't support it, so we ported the WSDL to document-literal, the result is in Axis2’s test harness – Benchmark.wsdl. After that we ported the BenchClient (the java class that makes the web services calls) to the toolkits we were testing (we used the latest SVN as of May 15, 2006 for Axis2 and used Axis 1.4). We ended up with three versions:

• Axis1 WSDL2Java generated code
• Axis2 WSDL2Java generated code with ADB data binding (Axis2's simple built-in databinding)
• Axis2 WSDL2Java generated code with Apache XMLBeans data binding

We also created versions of the benchmark services using Axis1,Axis2/ADB and Axis2/XmlBeans. We hosted the Axis1 war and the Axis2 war in a single instance of Apache Tomcat and the two Axis2 services (ADB and XMLBeans) were deployed into the same Axis2 webapp repository.

Test Environment

Summary of Results

The main findings are as follows:

• Axis2 (whether ADB or XMLBeans) is appreciably (between 4-5x) better than Axis1.
• In Axis2, ADB is marginally better than XMLBeans
• Axis2 beats Axis1 hands down for large data sets.

Since pictures say more than words... Here are the charts for first set of tests

The following charts depict the first scenario where both the client and service are using the same toolkit/databinding. You can download the Excel spreadsheet below to view other combinations/results. Legend: AXIS10 - Axis 1.X, AXIS2X - Axis2 with XmlBeans data binding, AXIS2A - Axis2 with ADB data binding

1.1 Echo Operation for primitives

Comment As you can see from this graph, the core performance on doubles, ints and Strings is a significant improvement. The Y-Axis of the graph is time taken, so lower is better. As you can see, the performance improvement is better with larger test sizes. In other words, Axis2 scales better for larger workloads.

1.2 Echo Operation for complexTypes

Comment This is the probably the key graph, as it shows the performance on complex types, which are the most common in real life. As you can see, once again Axis2 scales much better, with a linear profile, compared to Axis1 where the performance degrades as you scale. Also Axis2 shows much more consistent behaviour with the two complex types, whereas Axis1 has significant scaling issues with the SimpleEvent type.   This behaviour shows the effect that the redesign of Axis2 has had: Axis1 ends up using multiple copies of the message data, whereas Axis2 streamlines this to prevent copying between objects as much as possible. These numbers show the impact of that strategy.

Here are a further set of graphs you can click on.

Receive Operation for Primitives

Receive Operation for ComplexTypes

Send Operation for Primitives

Send Operation for complexTypes

Send/Receive/Echo Operations for Strings

Send/Receive/Echo Operation for ComplexTypes

Charts for mixed client/server

Obviously if you mix an Axis1 client with an Axis2 server (or vice-versa), the gains won't be so great. However, the following tests give some indication of the results.
Legend: AXIS10 - Axis 1.X, AXIS2A - Axis2 with ADB data binding (see X and Y legends)

2.1 Echo Operation for primitives

Comment This graph shows that although switching the client over from Axis1 to Axis2 helps, you get more benefit from switching the server over, even if you leave the clients using Axis1. Switching from an Axis1 server to an Axis2 server gives while leaving the client gives a rough 2x performance boost. Switching the client while leaving the server is slightly less effective.

Click here to see the results for Complex Types with Mixed Client/Server

What's the bottom line?

Just look at the time taken for running the whole test suite:

Axis2, without a doubt, offers much better performance compared to Axis 1.X, both as a service and as a client. When used at both ends, the performance gains can be significant, especially with large sets of complex types.

Please feel free to download the zip below, it has the source and scripts for the various clients and services, the logs from running the tests etc. Future directions for testing performance will include adding other stacks and JMeter based multi-threaded requests for measuring TPS and throughput.
If you have any questions/suggestions, please drop me an email to dims AT wso2.com.

Resources

• The Excel sheet(s) with a pivot chart (and data) for further analysis are available - perf1.xls perf2.xls
• Sources and logs for the client(s) and service(s) are here - benchmark-001.zip
• Axis2/Java Performance Testing Round #2 - Apache Axis2 vs. Codehaus XFire in a Contract First Scenario

Author

Devanum Sirinivas, WSO2 Inc. dims at wso2 dot com