An Analysis of Tools Used for WSO2 Test Cycles to Achieve Effective Results

  • By Shavantha Weerasinghe
  • 30 Jun, 2014

Table of contents

  • Introduction
  • Tools used for functional testing
  • Tools for performance testing
  • Tools used for profiling
  • Tools for security testing
  • Conclusion


WSO2’s competitive advantage lies in the leanness of its product offerings. The primary objective of the WSO2 Quality Assurance Team is to ensure the products delivered meet, and conform to requirements and fitness for use in delivering high quality products to its customers. To achieve these objectives, tools play a major role in helping to conduct effective testing of the products.

This article discusses, with practical scenarios, various testing tools used by the WSO2 Quality Assurance team during different stages of the product testing life cycle, explaining how each tool can be used.

WSO2 testing principles include the following:

  • Quality is the #1 priority engineering practice
  • Engineering owns and is responsible for quality
  • Everyone in engineering does testing
  • Collaborative testing methodology

Middleware is referred to software that facilitates exchange of data between two or more application programs within the same environment or different heterogeneous environments addressing common concerns ranging from enterprise applications that handle transforming and routing of messages to applications that manage authentication, authorization, and governance.

For a detailed understanding on an effective approach for testing of middleware refer to the article written by Asanka Vithanage [1].

At WSO2, we look at various factors when selecting the best tool for testing; some of the factors considered include

  • Cross platform compatibility
  • Open source tools
  • Support for Java
  • Time to learn the tool
  • Flexibility for testing need

Once a product is released for a test cycle, the WSO2 Quality Assurance Team performs various types of testing starting from verification and validation of the product feature and integration scenarios as well as evaluating the application against identified performance criteria to verifying if the application is secure on production environments.

Tools for functional testing


Functional testing of WSO2 products are performed using tools such as Apache JMeter, SoapUI as well as through command line tools, such as cURL and the use of simple Java Clients.

Let's first look at Apache JMeter; this is a 100% open source tool that supports both functional and performance testing activities. JMeter supports Web - HTTP, HTTPS,SOAP, FTP type of protocols.

Let's look at how this tool is used within WSO2 for functional testing purposes. Let's assume that in our test environment we have a WSO2 Enterprise Service Bus (ESB) hosting a simple pass through proxy service calling to a back-end service hosted on WSO2 Application Server.

To test the above-mentioned scenario for functionality, our objective should be to verify that the end-to-end flow works as expected for one thread.

With JMeter, the first task is to configure the Test Plan; within the test plan, we select the functional testing mode to record all the responses returned by the server. Next, we specify the Thread Group, where the number of request and the ramp up periods need to be configured. Next, to verify SOAP messages, we can use a sampler, such as SOAP/XML-RPC Request, where we specify the URL of the back-end service and the SOAP message to be sent.

Useful components of JMeter are

  • Test Plan: Helps to specify if the test should run as a functional test or not
  • Thread Group: Helps specify what load and ramp-up levels need to be used
  • Samplers: JMeter provides over 20 samplers that help generate test results for various use cases. One such sampler is the SOAP/XML-RPC request that helps generate SOAP requests to a web service
  • Listeners: A listener, such as the Aggregate Report, provides critical information regarding the test execution

Another important feature provided by this tool for functional testing is assertions. With assertions we can easily configure our expected results for each response received from the system the JMeter results will show (pass or failed) based on the response meeting the configured assertions.

For example, if we set an assertion to expect a HTTP 200 response, and the response is received to match the set assertion, then JMeter will update the test results as a pass.

When testing the functionality of an application, there are instances when we need to send multiple varied request. With JMeter, the concept of data driven testing can be achieved with the use of CSV data set configuration. With this element, we can call a CSV file with multiple values and then parameterize the data that is fed into the test scenario.

For example, instead of just passing a single value, as shown below, we can pass a parameter to which it will pass a new value for each iteration.


In production environments, there could be a think time of when a user initiates a task as well as delays that may occur due to network limitations and bandwidth usage; emulating these is vital to achieve accurate results. For our scenario, we can introduce these delays using a timer, such as the Constant Throughput Timer. Let’s say we want to introduce a 5 second delay; once set, JMeter at run time will introduce a variable pause between the request generated for each executed thread.

A full coverage is not within the scope of this article. However, for a detailed explanation on effective testing of service-oriented architecture refer to the following webinar [2] and article [3] listed under references.


Web services are one of the main building blocks of service-oriented solutions for enterprise applications where interoperability among heterogeneous applications can be achieved with SOAP and RESTful web services.

SoapUI is an open source functional testing tool that supports a range of protocols, such as SOAP, REST, HTTP, JMS, and helps in testing functional, performance and security aspects of an application and is used widely within the test cycles at WSO2. For this section, let's look at the use of SoapUI for functional testing of web services.

Take for example a scenario where we want to test the functionality of creating multiple secondary user stores [11] via a web service using WSO2 Identity Management server. For this task, the web service we use is UserStoreConfigAdminService?wsdl.

To perform functional testing for the UserStoreConfigAdminService?wsdl with SoapUI, we can use the concept of ‘test suites,’ which is one of the key building blocks of SoapUI. Within the suite, we can create a test case, which then enables us to pass the required parameters to the web service to create the user stores.

SoapUI also supports service mocking. When a web service is not yet fully implemented or if access is not available for a web service, then testing cannot be put on hold until implementation is completed and considering customer commitments. At WSO2, we use the concept of service mocking to ensure that test cycles are carried out to overcome such delays.

Coverage of all SoapUI functionality is not within the scope of this article; however, for a detailed explanation on effective testing of service-oriented architecture refer to the following webinar [2] and article [3] listed under references.


cURL is another tool used for conducting functional testing. This is a command line tool that helps transfer data from or to a server using protocols, such as HTTP, HTTPS, FTP as well as supports user authentication, FTP upload, HTTP post, and SSL connections.

Let's look at how cURL is used for functional testing purposes within WSO2. Take for example WSO2 API Manager; if we need to test an API in published status by invoking it, we may also need to invoke this API to simulate continuous flow of request. Once we have subscribed to the API and keys are generated, we use cURL to generate the request on the API under test.

Let’s take a scenario where we want to access all published APIs. To check this via cURL, we would need to first login to the API manager and then access the published list of APIs.

Curl Request to Login
curl -X POST -c cookies http://localhost:9763/store/site/blocks/user/login/ajax/login.jag -d 'action=login&username=admin&password=admin'
{"error" : false}shavantha@shavantha-ThinkPad-T530:~$


* connected
* Connected to localhost ( port 9763 (#0)
> POST /store/site/blocks/user/login/ajax/login.jag HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:9763
> Accept: */*
> Content-Length: 42
> Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 42 out of 42 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
 GET /store/site/blocks/api/listing/ajax/list.jag?action=getAllPublishedAPIs HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:9763
> Accept: */*
> Cookie: JSESSIONID=EB7769C3BF804B396A45265DDFF9EBDB
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported

JAVA Clients

Some test scenarios that we cover within WSO2 cannot be directly supported by third-party automation tools, and manual verification is not practical considering the time to release. In such an instance, in-built tools need to be used to cover the required test scenarios.

Let's assume that we need to check a scenario how WSO2 ESB works with a range of security policies. The situation would require different policies to be applied to secure a proxy service and to check if secure communication can be achieved with each to ensure end-to-end communication takes place as anticipated. With a Java client, these types of scenarios can be achieved accurately. To understand this type of scenario better, refer to articles [7] and [8], and the blog post [12], which will give you a clear understanding of the use of Java Clients for these situations.

Integration of a WSO2 product with another vendor product is another common occurrence at test cycles.

Sometimes these third-party vendor products may not be available at the time of product release to testing; for example, WSO2 Message Broker may need to integrate with another commercial JMS messaging product for publishing and subscription of messages.

Verification of these functionalities cannot be halted until the actual application is available for integration level testing; as a result, we can use Java Clients to act as message publishers or subscribers to integrate with the message broker. To understand this type of scenario better refer to the article [9], which gives you a detailed explanation of the use of Java Clients for these instances.

Tools for performance testing

Performance testing of middleware and service-oriented applications is a vital factor when meeting customer expectations. We need to ensure that products are capable of handling varied load levels at production and has a high fault tolerance capability.

The WSO2 products function in an integrated and sometimes clustered environment; as a result, load testing is a vital part of a testing carried out within the test cycles.

At the start of this article, we looked at how the JMeter tool can be used to carry out functional testing tasks. However, one of the main uses of this tool is to conduct performance and load testing on the middleware products.

Let's assume we want to test a load scenario where 300 threads are run with a ramp up of 5 minutes on a proxy deployed on WSO2 ESB calling a service deployed on WSO2 App Server; we want to know the throughput and latency values. With a tool like JMeter this can be easily configured by specifying the relevant configurations via thread groups.

Long running tests are conducted during the test cycles to help identify memory issues. When using the tool with UI, the memory consumed for running the UI may affect the output of the test. To avoid this, with JMeter, these long running test can be set easily by running the script in a headless nature. For more information on the use of JMeter for load testing refer to the webinar [2].

We have discussed how SoapUI provides support for service mocking. One main reason we use SoapUI in our test cycles is the flexibility the tool provides to conduct load test on these mock services. For a clear understanding of SoapUI a book published by WSO2’s Charitha Kankanamge is available for purchase at [4].

Tools used for profiling

Sometimes information provided by tools may not be adequate for analysis of a system while under performance test. Since WSO2 products are developed with Java, we may need to look at how a JVM behaves when load/stress is applied on an application at run time or by analyzing a heap dump. Profiling helps analyze at what stage an application performance suffers based on factors, such as memory usage, thread usage, etc.

At WSO2, we use two tools namely

Let's assume we want to test a load scenario where 300 threads are run with a ramp up of 5 minutes on a proxy deployed on WSO2 ESB calling a service deployed on WSO2 App Server; we want to know the throughput and latency values. With a tool like JMeter this can be easily configured by specifying the relevant configurations via thread groups.

  • The JProfiler (a commercially licensed Java profiling tool)
  • The JConsole (a built-in tool with the jdk that helps us to monitor an application’s performance remotely using a remote process or a locally running application using a local process)

For example, Let's assume the application under test keeps throwing OutOfMemoryErrors when the load is applied over a period of time. With a profiling tool, such as JConsole, we can monitor class count to determine if there are steady rises, or monitor the memory levels to determine if the results show an incremental climb. These results can help us to assume that perhaps the application may have a ClassLoader leak somewhere and will run out of PermGen space with the increase in load.

Apache Bench

Apache Bench is another tool used by the WSO2 team during performance and load testing cycles. This single-threaded command line based tool helps with tasks, such as measuring the performance of any web server.

Let's assume that we want to check the performance of a WSO2 product; the way to use this tool is by running a command similar to ab -n 100 -c 10 whereby the tool will provide an output with bench marking results similar to that shown below.

Server Software:    	WSO2-PassThrough-HTTP
Server Hostname:
Server Port:        	8280

Document Path:      	/carbon/
Document Length:    	0 bytes

Concurrency Level:  	10
Time taken for tests:   0.218 seconds
Complete requests:  	100
Failed requests:    	0
Write errors:       	0
Total transferred:  	11200 bytes
HTML transferred:   	0 bytes
Requests per second:	458.18 [#/sec] (mean)
Time per request:   	21.826 [ms] (mean)
Time per request:   	2.183 [ms] (mean, across all concurrent requests)
Transfer rate:      	50.11 [Kbytes/sec] received

Connection Times (ms)
          	min  mean[+/-sd] median   max
Connect:    	0	0   0.0  	0   	0
Processing: 	3   22  30.1 	12 	114
Waiting:    	3   21  30.1 	12 	114
Total:      	3   22  30.1 	12 	114

Percentage of the requests served within a certain time (ms)
  50% 	12
  66% 	15
  75% 	17
  80% 	22
  90%	107
  95%	109
  98%	111
  99%	114
 100%	114 (longest request)

Tools for security testing

When it comes to application security, some of the threats considered for security testing include

  • Spoofing
  • Phishing
  • Information disclosure

At WSO2 we give high priority to ensure that applications are secure. Prior to an application deployed in a production environment, we need to identify any vulnerabilities that the application may face for various threats.

SoapUI for security testing

Security testing tools are needed to scan for possible vulnerabilities. SoapUI is one of the tools we use for this purpose.

With SoapUI, by adding a test case for security testing, we are able to scan for possible vulnerabilities. Let's assume that we want to verify a web service for security vulnerabilities using SoapUI; by selecting the specific type of scan and the assertions to be performed through the Security Test Runner we can achieve the results for a scan.

JMeter for security testing

There are situations where we need to test a web service that does not support anonymous access; for such situations JMeter supports the concept of HTTP basic authentication.

When a web service is secured using HTTP basic authentication, then the user credentials are carried over HTTP headers of the message and we cannot use SOAP/XML-RPC request to directly send a message. As a result, in order to access a secured web service, the HTTP Authorization Manager can be used as discussed below.

  1. Insert the header into SOAP messages that transmit over an HTTP channel
  2. Add the Config element ‘HTTP Authorization Manager’ and specify the following details
  • Base URL = URL of the web service to test
  • username = username_value
  • password = password_value

Burp Suite

Another powerful tool used for security testing is the Burp Suite. This tool comes in two versions - the free edition and the professional edition, providing a variety of advanced features that we can use for security testing purposes. A detailed explanation of this tool is not within the scope of this article. However, discussed below are three of the main features available within the Burp Suite that are utilized during security testing cycles.

Let’s assume we need to login to the API manager, create an API, and subscribe and generate keys as our scenario.

When testing a management console of API manager, we may need to navigate to a page and create an API, publish the API, subscribe to the API, and log out. With Burp Suite, Spider we can create a sitemap of the application by recording all the requests the user performed while accessing the management console.

Attacks such as ‘man in the middle’ is when a request is intercepted, altered and sent as malicious content to a party that is expecting some data. To check how the application responses to altered request generated, with the Burp Repeater, we can alter a request and verify how well the application will handle such changes.

Let’s assume we need to check the API manager tool to create API and generate tokens. These tokens are used for invoking APIs. The sequencer is a feature that helps to analyze the degree of randomness in security-critical tokens issued by an application.

To download the tool and to obtain more information on the use of Burp Suite, visit the site at [5] and blog [6].


Using the right tool for the right tasks help to obtain effective results from test cycles. This article focussed on discussing the tools used within the WSO2 for its test cycles; their uses were explained in detail with scenarios on how we look at utilizing tools to achieve an efficient and effective test cycle.



About Author

  • Shavantha Weerasinghe
  • Senior QA Engineer
  • WSO2