Introducing Automated Tests for WSO2 Products
- Waruna Perera
- Associate Technical Lead - WSO2
Table of contents
- Introduction
- Creating an integration test module
- admin-clientss
- integration-test-utils
- test-artifacts
- Create base class
- Test case with inherited base class
- Creating test case module
- Writing test cases
- Managing dependencies
- Plugins and configurations
- Maven Surefire Plugin (maven-surefire-plugin)
- testing.xml
- testing-server-mgt.xml
- Maven Dependency Plugin (maven-dependency-plugin)
- Maven JAR Plugin
- Maven Surefire Report Plugin
- Maven Resources Plugin
- copy resource files
- artifacts.GREG directory
- automation.xml
- automationSchema.xsd
- emma.properties / filters. txt / instrumentation.txt
- testng-server-mgt.xml
- testing.xml
- As a whole package
- As a individual class
- Run test cases
- Summary
Introduction
Introducing automated tests to a WSO2 product can be done in few steps. WSO2 Automation Framework uses TestNG as its base and wraps its features around it. In this article we discuss how to introduce automated tests to a product. This article is not focused on “How to write test cases using TestNG”. The main objective is to make users understand how to configure dependencies, relevant pom files and automation related configurations. The WSO2 Governance Registry is used as the example product; you can find the source code here.
Creating an integration test module
First you need to create module for test cases. Create a new module under the modules folder in your product. the best practice is to create a hierarchical tree module. Following is an example structure:
- modules
- integration
- tests-common
- admin-clients
- integration-test-utils
- test-artifacts
- tests-integration
- test-module-1
- test-module-2
- dashboard
- distribution
- features
- login
- p2-profile-gen
- samples
- source
- styles
Here we have created a module called integration. Under integration there are sub modules. In automated test cases there are;
- tests cases which you need to run
- utils, client classes and test artifacts codes
Here is the part of pom.xml of the integration module:
org.wso2.governance governance-parent 5.0.0-SNAPSHOT ../../pom.xml 4.0.0 integration WSO2 Governance Registry - Integration Tests pom tests-common/admin-clients tests-common/integration-test-utils tests-common/test-artifacts tests-integration
We have created two sub modules under integration;
- tests-common
- tests-integration
The tests-common module is used to write common utilities for the test cases which are not provided by default with the automation framework. In the tests-common module there are sub modules. You should note that there is no pom.xml for tests-common module.
admin-clients
In this module we have written specific clients needed to run the test cases which are not provided at the automation framework supportive modules. This admin client module contains product specific admin service clients. There are a common set of clients available in the automation framework supportive modules as well. You can find them here.
integration-test-utils
This module contains common but specific utilities only for the product. Common utilities can be found here.
test-artifacts
This module is another utility module that is specific for the tests. You can keep code that is easy to debug or specific artifacts that need to be tested. We can't exactly direct which modules to be added as test-artifacts hence it is the choice of the test case developer. Some examples are webapps, registry handlers and custom mediators among others. You can find example code here.
Creating the base class for test cases
With this method it is easy to derive base methods that you need to write a test case. Here is the example base class. You should create this class in the test-utils module. In WSO2 GREG product we have created our base class in the integration-test-utils module.
package org.wso2.greg.integration.common.utils;
import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.wso2.carbon.automation.engine.context.AutomationContext; import org.wso2.carbon.automation.engine.context.TestUserMode; import org.wso2.carbon.automation.engine.frameworkutils.FrameworkPathUtil; import org.wso2.carbon.integration.common.utils.LoginLogoutClient; import javax.xml.xpath.XPathExpressionException; /** * Base class of all integration tests */ public class GREGIntegrationBaseTest { protected Log log = LogFactory.getLog(GREGIntegrationBaseTest.class); protected AutomationContext automationContext; protected String backendURL; protected void init(TestUserMode userMode) throws Exception { automationContext = new AutomationContext("GREG", userMode); backendURL = automationContext.getContextUrls().getBackEndUrl(); } protected void initPublisher(String productGroupName, String instanceName, TestUserMode userMode, String userKey) throws XPathExpressionException { automationContext = new AutomationContext(productGroupName, instanceName, userMode); backendURL = automationContext.getContextUrls().getBackEndUrl(); } protected String getBackendURL() throws XPathExpressionException { return automationContext.getContextUrls().getBackEndUrl(); } protected String getSessionCookie() throws Exception { LoginLogoutClient loginLogoutClient = new LoginLogoutClient(automationContext); return loginLogoutClient.login(); } protected String getServiceURL() throws XPathExpressionException { return automationContext.getContextUrls().getServiceUrl(); } protected String getTestArtifactLocation() { return FrameworkPathUtil.getSystemResourceLocation(); } }
Test case with inherited base class
public class SampleDataPopulatorTestCase extends GREGIntegrationBaseTest { @BeforeClass(alwaysRun = true, groups = {"wso2.greg"}) public void initTest() throws Exception { super.init(TestUserMode.SUPER_TENANT_ADMIN); }
Creating the test case module
When you write test cases that you need to run, first create a sub module under integration. This module should include test case sources and resource files. Following is its structure;
- test-module-1
- src
- test
- java
- resources
Writing test cases
Now its time to write your test cases. As we discussed earlier in the article you can simply derive it from your base class and continue. The pom.xml of your module will be as follows;
org.wso2.governance org.wso2.greg.integration.tests 5.0.0-SNAPSHOT ../pom.xml 4.0.0 Registry Extensibility Test Module org.wso2.carbon.registry.test.extensibility jar
Your test case module pom.xml should include all the dependencies and plugins that you need to run the tests. There are a few plugins that you should include here. Here is an example pom.xml. Lets look at the plugins that we use here and how to handle dependencies.
Managing dependencies
With products moving to GitHub you need to follow the exact guidelines to use dependencies in test cases. All the dependencies that you need to run the tests should be defined at the top level parent POM file with the version. In the given example dependency below you should note that the scope is defined as test;
org.wso2.carbon.automation org.wso2.carbon.automation.engine ${test.framework.version} test
In your test module POM, you should define the dependencies without version and scope. An example is shown below;
org.wso2.carbon.automation org.wso2.carbon.automation.engine
In your utility modules, the dependency scope should be overwritten. In the admin-clients module we have defined the dependencies with the scope compile. The reason for overwriting the scope is that we have defined utility modules as java classes compared with test classes in test modules.
org.wso2.carbon.automation org.wso2.carbon.automation.engine compile
You can learn more about dependency scopes here.
Plugins and configurations
Maven Surefire Plugin(maven-surefire-plugin)
This is the plugin that is used to run the test cases. It runs the unit tests of the product. You should note that this plugin generates test reports in plain text and xml formats. To add more usability to the testing by generating reports in HTML format we use Maven Surefire Report Plugin which will be described later.
Under the configurations that belong to this plugin first we have defined the test cases that need to be run. You should note that there are two configuration files stated here;
src/test/resources/testing-server-mgt.xml src/test/resources/testing.xml
testing.xml
This describes the test cases that need to run.
testing-server-mgt.xml
This describes any servers or services that you need to start before running the tests.
To run the tests you need to run the carbon server. To do that we have specified the location of product distribution as a property.
carbon.zip ${basedir}/../../../distribution/target/wso2greg-${project.version}.zip
We define resources location under this plugin as a property.
framework.resource.location ${basedir}/src/test/resources/
Since we use a wrapped set of listener classes, the default TestNG listener classes should be disabled with the following property;
usedefaultlisteners false
For code coverage functionality we have defined the following properties as well. The test framework looks for these emma related properties when instrumenting jars and generating coverage reports.
${basedir}/target/emma ${basedir}/src/test/resources/instrumentation.txt ${basedir}/src/test/resources/filters.txt ${basedir}/target/emma
Maven Dependency Plugin (maven-dependency-plugin)
This plugin is used to copy and unpack necessary files and artifacts. Following is a code snippet of it;
copy-emma-dependencies compile copy-dependencies ${project.build.directory}/emma jar emma,org.wso2.carbon.automation.core
Maven JAR Plugin
This plugin is used to build jars.
org.apache.maven.plugins maven-jar-plugin 2.4 test-jar
Maven Surefire Report Plugin
This plugin is used to generate html and xml based reports of tests that were run. After you run the test cases, inside the target folder of your test module there will be a folder created for surefire-reports.
org.apache.maven.plugins maven-surefire-report-plugin 2.7.1 ${basedir}/target/report registry-api-test integration-test report-only
Maven Resources Plugin
This plugin is used to copy resources to the output directory. We copy only the test resources here. When you go through execution plans under this plugin, there are several tasks that have been done. The plugin copies resources from the product pack which resides in the local maven repository. You must remember that it doesnt copy resources from the product pack which resides in the distribution folder that you have built.
- copying keystores
- copying axis configs
- copying client modules
Copy resource files
- resources
- artifacts
- GREG
- wsdl
- schema
- policies
- aar
- axis2config
- keystores
- products
automation.xml automationSchema.xsd emma.properties instrumentation.txt filter.txt testing.xml testng-server-mgt.xml
Here we will look at must need resources individually and understand what they do.
artifacts.GREG directory
In the artifacts/GREG folder we need to include resource files such as aar, car and wsdl. Test cases will use this resources in run time.
automation.xml
This includes the configurations that we will need to run the test cases. The example provided will have the all the configurations described here. All the elements are explained in the automation.xml. There are some parameters you need to understand. All the other parameters are self explanatory.
The WSO2 Automation Framework provides in-built TestNG listener classes. You need to define which classes you are going to use in order to run the test suites.
org.wso2.carbon.integration.common.extensions.carbonserver.CarbonServerExtension org.wso2.carbon.integration.common.extensions.usermgt.UserPopulateExtension
CarbonServerExtension class does the initial configuration background work to run the test cases that are written in the automation framework. This class starts the carbon server on the configured port and does other necessary configurations like jar instrumentation. You can review the source here.
UserPopulateExtension class does the population of users to run the test cases. Note that you need to define those uses in the automation.xml file.
automationSchema.xsd
This is the schema file of automation.xml
emma.properties / filters. txt / instrumentation.txt
These files are used for code coverage. For example, files can be found at the source code location.
testng-server-mgt.xml
This file contains all the in-built TestNG listeners provided by test framework. You can define any other pre-sequences that should be run before the start of the test cases in this file.
testing.xml
This is the configuration file where you need to specify which test cases you need to run and which listener classes to use.
First you need to define listener classes that are used to run the tests.
Then you need to define your test cases. There are two methods to define it;
As a whole package
If you define it as a package, all the test cases in the package will be executed.
As an individual class
This is really useful if you want to identify an issue in failed test cases. Instead of running all the test cases, you can individually run each test case.
Run test cases
mvn clean install : run the test cases
mvn clean install -Dmaven.surefire.debug : debug test case
Summary
To introduce automation to WSO2 products you only need to have a basic knowledge of TestNG. Configuration is all you have to do to run test cases successfully.