The Use of Productivity Acceleration Tools for SOA Testing

  • By Nirodha Gallage
  • 15 Oct, 2013

Table of contents


Advantages of using tools for SOA testing

Below are some of the advantages of using tools for your SOA solution testing.

  • There are many proprietary tools available in the market and many free, open source tools are readily available for download. Without unnecessarily spending time on writing new tools from scratch, one can reuse the tools out there, which will inturn enable you to invest the time saved on a more important task. 
  • Manual testing is time consuming. In situations where you have frequent release cycles, which involves recursive testing, usage of tools are very important. In such releases, you are given a limited period of time for testing, and therefore, using automation tests accelerates your testing process rather than doing them manually.
  • Assume that there are product experts or feature experts. In every release cycle, if we get these experts to execute the basic test cases manually, it would be a waste. Instead of doing so, if we automate those scenarios, the experts can skip testing them manually and utilize their knowledge to dig deep into the product and carry out exploratory testing.
  • One of the key concepts of SOA is to release early and release often. In such scenarios, if you plan to execute all your test scenarios manually, you will not be able to cater to this. Assume that you have thousands of test cases to be executed within a week, running them manually will take more time than you predict and you are repeating the same task continuously. If these features are automated, testers can skip them and move to platform level testing, performance testing, etc.
  • When you manually run test cases, there is a possibility for human error and for you to miss out scenarios. However, if all these scenarios are automated, we can ensure that all of them will run without being missed out and human error will also be minimized.
  • There are tools out there that support dynamic data driven testing, which means they generate dynamic requests with random data. So you can run your automated tests with different data similar to manual testing.
  • One of the biggest advantages of using tools for testing is that you can get comprehensive test reports generated. You can let a performance test or even functional tests to execute and get a detailed report once completed. This is really convenient because if we do manual testing, we have to manually generate the report ourselves.
  • Tools like SoapUI and JMeter have the ability to run tests in headless mode. Therefore, one can automate the scenarios and integrate them with the build systems; while the product is being built, half of the testing will be completed.


Choosing the right tool for your SOA solution

Of the many tools available in the market, it is important to choose the best one to suit your needs. Below are some points you can take into consideration when selecting the right tool.

  • Ability to generate request messages automatically - assume you have web services in your SOA solution and their service contracts are readily available. So when the service contract (?wsdl) is provided, it will automatically generate the request messages for you. SoapUI provides this feature.
  • Validating responses using assertions - when you run automated tests, there is no point in validating each and every response message manually. This process too should be automated. SoapUI as well as JMeter provide the ability to assert responses.
  • QoS enabled service invocation - in your SOA solution, there may be services that facilitate QoS properties such as WS-Security, WS-Addressing, MTOM, etc. Most tools out there do not support such properties; however, SoapUI will provide you the facility to invoke any service secured with WS-Security policies or services with MTOM enabled, etc.
  • Service simulation - assume your final SOA solution is not complete yet, but you have the service contracts readily available. So without having to wait until the last moment, you can use the service contracts and simulate the backend services and test them out. SoapUI provides a solution for this by allowing you to create mock services to simulate the real services.
  • Support for multiple transport protocols - your SOA solution might be dealing with different types of transport protocols such as HTTP, HTTPS, JMS, etc. As a solution, you can use both SoapUI and JMeter since both tools support these different transport protocols.
  • Multiple message support - if your solution deals with different types of messages, i.e. SOAP, POX, JSON, REST, both SoapUI and JMeter will serve your purpose as both of these tools can be used to send the aforementioned request types. 

Example scenarios 

Web Testcase sample

The Web Testcase method allows you to create projects without a service contract (A ?wsdl is not necessary). With this method, you can do simple CRUD operations, i.e.GET, PUT, POST, etc. In other words, you can do RESTful invocations through this method. An advantage of using this method is that you can have test cases of multiple services within the same Web Testcase project.


1. Select File > New SoapUI Project and it will open up the New SoapUI Project window. 
2. Specify a project name and tick the checkbox Create Web Testcase. Click OK.
3. When it opens up the Add Web Testcase window, specify the web address that you want to invoke, e.g. https://localhost:9763/services/AccountService
4. Specify a suitable value as the Web Testcase Name.
5. Select the preferred Target TestSuite from the dropdown.
6. Tick off the Start Recording immediately checkbox and click OK.
7. When the request editor window opens, select the HTTP method you want to invoke, specify the query parameters/specify the request message and invoke the service.

8. Once invoked, you will see the successful response through the Response tab as below.

Invoke a secured web service using basic authentication

If your service is secured and challenges basic authentication details from the client, there are many ways such a service can be invoked using SoapUI. We will be discussing two such methods in this article. They include setting basic authentication credentials
a) on each SOAP test request
b) through Interface Viewer

Setting basic authentication credentials at SOAP request level

1. Open up the Request Editor.
2. Click on the button labeled Aut at the bottom of the request editor window.
3. Specify the username and password.
4. Select Preemptive or Global HTTP Settings from the Authorisation Type dropdown and then send a request.
Note: If you select Global HTTP Settings, make sure the checkbox Authenticate Preemptively is ticked through SoapUI Global Preferences window.

Through this method, basic authentication properties will be set for a particular request only. If you have multiple requests, you will need to set these properties for each and every request message.

How to set basic authentication credentials through Interface Viewer

1. Right-click on a WSDL operation in the project explorer tree on the left-hand side of the SoapUI interface and choose Show Interface Viewer.
2. Select the tab Service Endpoints.
3. Then specify values for the Username and Password fields and invoke the service.

When you send a requests with basic authentication credentials, through the raw view of the message, you will see credentials being passed as an HTTP header.


POST https://localhost:9443/services/Axis2Service.Axis2ServiceHttpsSoap11Endp... HTTP/1.1

Accept-Encoding: gzip,deflate

Content-Type: text/xml;charset=UTF-8

SOAPAction: "urn:echoString"

Content-Length: 333


Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

Authorization: Basic YWRtaW46YWRtaW4=

<soapenv:Envelope xmlns:soapenv="" xmlns:ser="">




        <ser:s>Hello World!!!</ser:s>



Invoke a web service secured with asymmetric binding policies

By signing SOAP messages, we ensure integrity of the message, where we can confirm that the message is not tampered with while being transferred between the sender and the receiver. This also ensures non-repudiation of messages where it guarantees that neither the sender nor the receiver can deny that the message was not received or sent. Signing a SOAP message involves the following steps.

  • Create a digest value of the message to ensure the integrity of the message
  • Encrypt the digest value from the private key of the client
  • Client sent the request with the encrypted digest value
  • Once received by the server, decrypt the encrypted digest using the public key of the client
  • Server creates the digest value of the received message to ensure whether the generated hash value is similar to the decrypted digest value

To sign messages, you need keystores. In this example, assume that we have already created a pair of public/private keys. You can download them here.


Assume that your service is secured using an asymmetric binding policy. Using the ?wsdl of the service, create an SoapUI project. Invoke it once and it will return SoapFault with the message Missing wsse:Security header in request, which means the service is expecting a secured message. So let us see how we can send a signed message using SoapUI.

Uploading the keystore 

1. Right click on the project you created above and select Show Project View.
2. In our scenario, both the client’s private key and the server’s public key are stored in the client.jks. Therefore, select the WS-Security Configurations tab, then from there, select the tab Keystores.
3. Click on the + icon and browse for the clientks.jks from your file system and specify the password as clientks.
4. Specify clientks as the Default Alias and Alias Password. If the keystore is added successfully to the project with THE correct keystore password, the status will be shown as OK.

Configuring outgoing WS-Security properties  

1. Click on Outgoing WS-Security Configurations tab.
2. Click on + icon and specify SignOutgoingConfig as the name for the outgoing configuration and leave the rest of the fields blank.
3. Then click on the + icon at the bottom to add WSS configurations. Select Signature from the dropdown.
4. Select/set the below mentioned values.
        Keystore - clientks.jks
        Alias - clientks
        Password - clientks
        Key Identifier Type - Binary Security Token or X509 certificate
        Signature Algorithm - or set it as default
        Signature Canonicalization - Leave as default
        Digest Algorithm -
        Use Single Certificate - Tick this checkbox
                    Parts - ID : Leave empty
                    Name : Body
                    Namespace : or (depending whether the message is soap11 or soap12)
                    Encode : Content
5. Next, click on the + icon again and add another WSS configuration which is Timestamp. Specify the Time To Live value as 60 seconds.

Configuring incoming WS-Security properties

1. Select the tab Incoming WS-Security Configurations and click on + to add a incoming WS-Configuration. Specify the name as SignOutgoingConfig.
2. Click on the + icon to add a new WSS configuration.
3. Select clientks.jks as the Decrypt Keystore and the Signature Keystore.
4. Specify clientks as the Password.

Applying WS-Configuration to the SOAP request

Once the above configuration is done, we need to set the incoming and outgoing WS-Security configurations to the SOAP request.

1. Select the echoString request and open the request editor window.
2. Click on the Aut button at the bottom of the window.
3. Select the Outgoing WSS as SignOutgoingConfig and Incoming WSS as SignIngoingConfig.
4. Invoke the service and you will get the expected response with a signature.
When you view the raw view of the request, you will see WS-Security headers being passed with the request.


Accept-Encoding: gzip,deflate

Content-Type: text/xml;charset=UTF-8

SOAPAction: "urn:echoString"

Content-Length: 3181


Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

<soapenv:Envelope xmlns:ser="" xmlns:soapenv="">


<wsu:Timestamp wsu:Id="TS-11">


















Note: When you look through the raw view of the request message, you might see only a small part of the message. This is because SoapUI can control the size of the message to be displayed here. This is configurable through the UI settings of Global Preferences UI.


Assertions for responses

Assertions can be used to validate responses received by a Teststep once it’s executed. A comparison is made between the parts of the response message (or the entire message) to some expected value. SoapUI allows to assert the content of the response message, HTTP status codes, SOAP faults, etc.

Assertions are added to Teststeps as shown below.

Once you click on the above highlighted Assertions label, it will open up the Add Assertions window where you can select which type of assertion you want to add to your teststep.

Sending chunk encoded messages

Chunked encoding is useful when a large amount of data is being returned to the client and the total size of the response may not be known until the request has been fully processed. An example of this is the generation of an HTML table of results from a database query. Send a request to your service as explained above and open the raw view of the message and you will see that the message is not transferred in chunks. You will see the content length of the entire message being passed as shown below.



Accept-Encoding: gzip,deflate

Content-Type: text/xml;charset=UTF-8

SOAPAction: "urn:echoString"

Content-Length: 309


Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)




        <ser:s>Hello World!!!</ser:s>



To get chunk encoded responses, we need to set the Content-Length Header with the size of the chunk as the value as shown below.

Send a request once again and see the HTTP headers through the raw view of the request message. Notice the HTTP header Transfer-Encoding. This header will indicate that the message is being passed in chunk  

Execute SoapUI tests in headless mode

There can be scenarios where you will have to run SoapUI from a machine where there is no user interface. In such situations, there needs to be a way to run a test without a UI. SoapUI provides external runners to execute unit tests and load tests in a headless environment. It gives you the ability to run either an entire test suite or individual test cases.

If you are on Windows, you can run the script as below.

testrunner.bat -s Axis2Service_TestSuite -r -f C:\soapUI\logs\response -l C:\soapUI\projects\my_soapui-project.xml

If you are on a Unix environment, the command would be as below.

./ -s Axis2Service_TestSuite -r -f /opt/soapUI/logs/response -l /opt/soapUI/projects/my_soapui-project.xml 

Below are some of the parameters that you can pass.

                  s - the test suite to run
                  c - the test case to run
                  r - prints a summary report
                  l - ignore errors and continue
                  f - folder to which test results should be saved 

In which situations can we use JMeter?

  • To simulate multiple user access when doing load tests
  • For random data driven testing
  • To analyze and measure performance of services
  • To validate responses, i.e. assertions
  • To create test suites with real-world scenarios that involve executing multiple services sequentially or in parallel
  • Testing for different transport protocols, i.e. HTTP, HTTPS, JMS, MAIL, FTP, JDBC, LDAP etc.


JMeter Example 1: Testing a web service with random data inputs

In this example, we are going to see how to invoke an Axis2 Web service with JMeter as described in the steps below (test script file can be found as StockQuote_WS.jmx)

1. Firstly, for all JMeter test plans, you need to add a Thread Group as the top element. Right Click on Test Plan > Add > Thhreads (Users) > Thread Group. There you can specify the number of concurrent users accessing the service, Ramp up time, and the number of loops etc.

2. Then Right Click on Thread Group and then select Add > Sampler >SOAP/XML-RPC Request. Here you have to specify the service endpoint URL and the SOAP request message and in this example, SimpleStockQuote service that is hosted on WSO2 Application Server has been used (you can generate the SOAP request using the SoapUI tool by providing the WSDL of the service). Also, note that the input “symbol” is randomized using the variable called ${company}.

<soapenv:Envelope xmlns:soapenv="" xmlns:ser="https://services.samples" xmlns:xsd="https://services.samples/xsd">










3. As we want to randomize the data inputs provided to the service, we use a CSV file (StockQuote_inputs.csv) to store the input data. In JMeter, ‘CSV Data Set Config Element’ is used to specify that CSV file and other parameters. Right click on the thread group > Add > Config Element > CSV Data Set Config. There you need to specify the path to that .csv file. Variable reference name (“company”), delimiter etc.

4. Now the basic test script is ready. If you run this test plan now you will be able to send the requests to the Stock Quote Service. But we cannot see the requests or responses yet. For that, we need to add a results tree component to see the responses of the requests. Right click on Thread Group > Add > Listener > View Results Tree. There you can see the requests sent and the responses, response codes, etc. for those.

5. Now let’s say we need to increase the thread count and do a load test. In that case, it is not practical to go through all the responses and check whether they are correct. For this case, we can define an assertion to be checked in the response and to mark the responses in red that fail that assertion, in View Results Tree. In this case, I have set a string value in the response as the assertion. Right Click on Thread Group > Add > Assertions > Response Assertion.

6. Finally, when doing load tests we need to know the average latency, throughput, etc. To get those values calculated you can use the Aggregate Report component. Right Click on Thread Group > Add > Listener > Aggregate Report.


JMeter Example 2: Get an Access Token from an OAuth2 endpoint and then use that to invoke an API

This example is a bit advanced and more components are used other than the basic components used in Example 1. Additional components used are described below. In this scenario, a REST call is sent to an OAuth2 web service via an API to acquire an access token. Then the response will be read to extract the access token from the response and put into a variable. Thereafter, that access token is read through that variable and used in the next API invocation (refer the test script ‘API_GAteway.jmx’ given in samples.)

1. Add a Thread Group. Right Click on Test Plan > Add > Threads (Users) > Thread Group.

2. We use a ‘User defined variables’ component to store the values such as server hostname and ports in variables. The advantage of doing this is when these values are set in several places you can refer to the variable and get the value. Then, later, if you want to change the values (e.g. change the endpoints from the QA environment to the Production environment) you can change it in this component in a central place. Right Click on Thread Group > Add > Config Elements > User Defined Variables.

3. Then, we need to do a HTTP POST call to ‘Token API’ in WSO2 API Manager. For that, we need to use “HTTP Request” component in JMeter. Right Click on Thread Group > Add > Sampler > HTTP Request

4. Also, we need to send some HTTP headers with that request so you need to add the “HTTP Header Manager” component as well. There you can add the header names and the respective values separately. For this request there are two headers set, one is ‘Content-Type’ and the other is ‘Authorization’ header, which is used to pass the consumer key-secret pair encoded in BASE64. Refer to this documentation for additional information on accessing the token API.

5. After we get the response from calling the Token API we need to extract the access token from the response and store it in a variable. Then we can refer to the token using that variable and set it in a header when calling the actual API later. For that, we need to add the “Regular Expression Extractor” component to the request. There you can specify the regular expression to extract the value of the “access_token” from the JSON response, and also a variable name to store the token in. Right Click on HTTP Request > Add > Post Processors > Regular Expression Extractor.

6. Also, you can add a response assertion to the same request to assert whether we got the correct response.

7. In this scenario, we need to invoke the ‘token API’ only once to get the access token and need to use that token to invoke the subscribed API multiple times. So we need to run the API invocation many times inside a loop after acquiring the token. For that, we add “Loop Controller” and specify how many times we need to run the loop. Moreover, add the next API invocation call inside that loop. Right click on Thread Group > Add > Logic Controller > Loop Controller

8. Then we need to send a HTTP POST request as an API call. Since we are invoking the same backend service via an API we have to use the same SOAP request message we used in Example 1. The access URL of the API should be given as the endpoint. Here too we need to send some HTTP headers with the POST request. Since we are sending an XML body, we need to set the Content-Type header. Also, as we need the access token to invoke the API we require to send that in the Authorization header. In addition, we have to use a response assertion that is the same as in Example 1 to verify the response.

9. Again we use the View Results Tree component and the Aggregate Report component to check the responses and other performance values etc. same as in Example 1.

Example 3: Running JMeter Scripts in Headless mode

This is a very valuable feature in JMeter where you can run the script without running the JMeter with UI. The advantage of this is that you can run JMeter in where there is only a command line console, or in a remote server machine. After you create the script using the UI it is saved as an XML configuration file with the extension of ‘.jmx’. Then you can copy this file to any machine that has JMeter and a compatible JRE installed.

To run the script in Headless mode you have to run the (or jmeter.bat in Windows) with the following parameters.

  • Linux : ./ -n -t /path/to/JmeterTest.jmx -l /path/to/resultsFile.jtl

  • Windows : jmeter.bat -n -t \path\to\JmeterTest.jmx -l \path\to\resultsFile.jtl

  • Another advantage is that you can change the test parameters programmatically using a shell script of batch script and execute the test plans recursively, as explained in this blog.



    • When sending http requests with JMeter always select 'httpclient4' as the implementation for high load scenarios.
    • When doing load tests disable "View Results Tree" as it affects thread creation rate.
    • Always keep test parameters like endpoints, IPs, ports etc. as configurable variables.
    • For high load tests, run the JMeter tests in headless mode, and write the result to a .jtl file. You can open them later in JMeter UI, to check results summary, and performance figures.
    • When measuring performance figures with load tests make sure only the client and server are running in their respective machine. 


Performance and security are some of the main areas of SOA solutions that needs to be thoroughly tested. Performing manual testing in those complex areas can be cumbersome and time consuming for tasks lilke generating request messages or writing service clients from scratch. By introducing tools to your testing process you can easily cut down time spent on the above areas and most of those can be automated and executed recursively. There are many proprietary as well as free tools available for testing, and this article provides examples for two of those open source tools: SoapUI and Apache JMeter. With the use of these tools wherever necessary you can create test suites and recursively execute them easily to test your solutions, which is useful when there are frequent builds or updates to your solution. It also helps to utilize your best-experienced testers to do exploratory testing and write test scripts for those areas and, thereafter, provide those scripts to execute recursively for another less-experienced one, and move on with another testing area. In many ways, as explained, tools can play a major part in your QA process.

About Author

  • Nirodha Gallage
  • Senior Software Engineer
  • WSO2