WSO2Con 2013 CFP Banner

Axis2/Java – Performance Testing Round #1

Discuss this article on Stack Overflow
By Davanum Srinivas
  • 17 May, 2006
  • Level: 
  • Reads: 23095

Some say that there are more Web Service benchmarks than there are Web Services protocols. And that's plenty! Everyone has their favorite set of tests and so far there isn't a single agreed standard benchmark. For example: Sun originally published a white paper Comparing JavaTM 2 Enterprise Edition (J2EETM platform) and .NET Framework. Microsoft responded with its own paper Comparing Web Service Performance, which caused Sun to respond. We decided to take a different tack for testing Axis2 performance...

Davanum Srinivas

WSO2 Inc.

Test Suite

We decided to follow a set of academic work that has been done in this area. We started off with Alek and Madhu’s “SOAP Benchmark Suite v1” and “SOAP Benchmark Suite v2”. The original SOAP Bench v1 had send/receive/echo operations for arrays of bytes (BASE64 encoded), integers, doubles, and strings. SOAP Bench v2 added send/receive/echo operations for two more data structures: SimpleEvent (a complexType with a sequenceNumber/integer, timestamp/double, message/string) and MeshInterfaceObject (a complexType with 2 integers x, y and a double value). Please see details on the v1 test suite here - grid2004.pdf and the v2 test suite here - soapBench-SC05.pdf.

Deciding what to test and how

Once we decided on the benchmark suite we came across an issue: the benchmark WSDL is RPC-Encoded. WS-I BaseProfile 1.0 bans RPC-Encoded, and Axis2 doesn't support it, so we ported the WSDL to document-literal, the result is in Axis2’s test harness – Benchmark.wsdl. After that we ported the BenchClient (the java class that makes the web services calls) to the toolkits we were testing (we used the latest SVN as of May 15, 2006 for Axis2 and used Axis 1.4). We ended up with three versions:

  • Axis1 WSDL2Java generated code
  • Axis2 WSDL2Java generated code with ADB data binding (Axis2's simple built-in databinding)
  • Axis2 WSDL2Java generated code with Apache XMLBeans data binding

We also created versions of the benchmark services using Axis1,Axis2/ADB and Axis2/XmlBeans. We hosted the Axis1 war and the Axis2 war in a single instance of Apache Tomcat and the two Axis2 services (ADB and XMLBeans) were deployed into the same Axis2 webapp repository.

Test Environment

The server was run on a Tomcat instance (Apache Tomcat/5.5.17) using JDK1.5 (build 1.5.0_06-b05) on Ubuntu Dapper (Linux version 2.6.15-18-686)and the machine specification is as follows:
Dell Dimension 9150 Pentium(r) D Processor 820 with Dual Core Technology (2.80GHz, 800FSB), with 2GB Dual Channel DDR2 SDRAM
 
Tomcat was run with the following JAVA_OPTS:
-server -Xms256m -Xmx700m -Xincgc -XX:PermSize=128m -XX:MaxPermSize=256m
 
The clients were run on a Lenovo Thinkpad T43 using JDK1.5 (build 1.5.0_06-b05) on Windows XP Service Pack 2
Lenovo ThinkPad T43 Intel Pentium M processor 760 (2.00GHz), with 1.5 GB SDRAM

Summary of Results

The main findings are as follows:

  • Axis2 (whether ADB or XMLBeans) is appreciably (between 4-5x) better than Axis1.
  • In Axis2, ADB is marginally better than XMLBeans
  • Axis2 beats Axis1 hands down for large data sets.

Since pictures say more than words... Here are the charts for first set of tests

The following charts depict the first scenario where both the client and service are using the same toolkit/databinding. You can download the Excel spreadsheet below to view other combinations/results. Legend: AXIS10 - Axis 1.X, AXIS2X - Axis2 with XmlBeans data binding, AXIS2A - Axis2 with ADB data binding

1.1 Echo Operation for primitives

Comment

As you can see from this graph, the core performance on doubles, ints and Strings is a significant improvement. The Y-Axis of the graph is time taken, so lower is better. As you can see, the performance improvement is better with larger test sizes. In other words, Axis2 scales better for larger workloads.

 

1.2 Echo Operation for complexTypes

Comment

This is the probably the key graph, as it shows the performance on complex types, which are the most common in real life. As you can see, once again Axis2 scales much better, with a linear profile, compared to Axis1 where the performance degrades as you scale. Also Axis2 shows much more consistent behaviour with the two complex types, whereas Axis1 has significant scaling issues with the SimpleEvent type.

 

This behaviour shows the effect that the redesign of Axis2 has had: Axis1 ends up using multiple copies of the message data, whereas Axis2 streamlines this to prevent copying between objects as much as possible. These numbers show the impact of that strategy.
Here are a further set of graphs you can click on.
Receive Operation for Primitives
Receive Operation for ComplexTypes
Send Operation for Primitives
Send Operation for complexTypes
Send/Receive/Echo Operations for Strings
Send/Receive/Echo Operation for ComplexTypes

Charts for mixed client/server

Obviously if you mix an Axis1 client with an Axis2 server (or vice-versa), the gains won't be so great. However, the following tests give some indication of the results.
Legend: AXIS10 - Axis 1.X, AXIS2A - Axis2 with ADB data binding (see X and Y legends)

2.1 Echo Operation for primitives

Comment

This graph shows that although switching the client over from Axis1 to Axis2 helps, you get more benefit from switching the server over, even if you leave the clients using Axis1. Switching from an Axis1 server to an Axis2 server gives while leaving the client gives a rough 2x performance boost. Switching the client while leaving the server is slightly less effective.
Click here to see the results for Complex Types with Mixed Client/Server

What's the bottom line?

Just look at the time taken for running the whole test suite:

Axis2/ADB Client - Axis2/ADB Service 178.829 seconds
Axis2/XmlBeans Client - Axis2/XmlBeans Service 330.093 seconds
Axis1.X Client - Axis2/ADB Service 421.219 seconds
Axis1.X Client - Axis1.X Service 1359.094 seconds

Axis2, without a doubt, offers much better performance compared to Axis 1.X, both as a service and as a client. When used at both ends, the performance gains can be significant, especially with large sets of complex types.

Please feel free to download the zip below, it has the source and scripts for the various clients and services, the logs from running the tests etc. Future directions for testing performance will include adding other stacks and JMeter based multi-threaded requests for measuring TPS and throughput.
If you have any questions/suggestions, please drop me an email to dims AT wso2.com.

Resources

Author

Devanum Sirinivas, WSO2 Inc. dims at wso2 dot com

WSO2Con 2014 USA