Introducing Automated Tests for WSO2 Products

Table of contents

• Introduction
• Creating an integration test module
• admin-clientss
• integration-test-utils
• test-artifacts
• Create base class
• Test case with inherited base class
• Creating test case module
• Writing test cases
• Managing dependencies
• Plugins and configurations
• Maven Surefire Plugin (maven-surefire-plugin)
• testing.xml
• testing-server-mgt.xml
• Maven Dependency Plugin (maven-dependency-plugin)
• Maven JAR Plugin
• Maven Surefire Report Plugin
• Maven Resources Plugin
• copy resource files
• artifacts.GREG directory
• automation.xml
• automationSchema.xsd
• emma.properties / filters. txt / instrumentation.txt
• testng-server-mgt.xml
• testing.xml
• As a whole package
• As a individual class
• Run test cases
• Summary

Introduction

Introducing automated tests to a WSO2 product can be done in few steps. WSO2 Automation Framework uses TestNG as its base and wraps its features around it. In this article we discuss how to introduce automated tests to a product. This article is not focused on “How to write test cases using TestNG”. The main objective is to make users understand how to configure dependencies, relevant pom files and automation related configurations. The WSO2 Governance Registry is used as the example product; you can find the source code here.

Creating an integration test module

First you need to create module for test cases. Create a new module under the modules folder in your product. the best practice is to create a hierarchical tree module. Following is an example structure:

• modules
• integration
• tests-common
• admin-clients
• integration-test-utils
• test-artifacts
• tests-integration
• test-module-1
• test-module-2
• dashboard
• distribution
• features
• login
• p2-profile-gen
• samples
• source
• styles

Here we have created a module called integration. Under integration there are sub modules. In automated test cases there are;

• tests cases which you need to run
• utils, client classes and test artifacts codes

Here is the part of pom.xml of the integration module:

  

        org.wso2.governance
        governance-parent
        5.0.0-SNAPSHOT
        ../../pom.xml
    

    4.0.0
    integration
    WSO2 Governance Registry - Integration Tests
    pom

    
        tests-common/admin-clients
        tests-common/integration-test-utils
        tests-common/test-artifacts
        tests-integration
    

We have created two sub modules under integration;

• tests-common
• tests-integration

The tests-common module is used to write common utilities for the test cases which are not provided by default with the automation framework. In the tests-common module there are sub modules. You should note that there is no pom.xml for tests-common module.

admin-clients

In this module we have written specific clients needed to run the test cases which are not provided at the automation framework supportive modules. This admin client module contains product specific admin service clients. There are a common set of clients available in the automation framework supportive modules as well. You can find them here.

integration-test-utils

This module contains common but specific utilities only for the product. Common utilities can be found here.

test-artifacts

This module is another utility module that is specific for the tests. You can keep code that is easy to debug or specific artifacts that need to be tested. We can't exactly direct which modules to be added as test-artifacts hence it is the choice of the test case developer. Some examples are webapps, registry handlers and custom mediators among others. You can find example code here.

Creating the base class for test cases

With this method it is easy to derive base methods that you need to write a test case. Here is the example base class. You should create this class in the test-utils module. In WSO2 GREG product we have created our base class in the integration-test-utils module.

package org.wso2.greg.integration.common.utils;

  
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.wso2.carbon.automation.engine.context.AutomationContext;
import org.wso2.carbon.automation.engine.context.TestUserMode;
import org.wso2.carbon.automation.engine.frameworkutils.FrameworkPathUtil;
import org.wso2.carbon.integration.common.utils.LoginLogoutClient;

import javax.xml.xpath.XPathExpressionException;

/**
 * Base class of all integration tests
 */
public class GREGIntegrationBaseTest {

    protected Log log = LogFactory.getLog(GREGIntegrationBaseTest.class);
    protected AutomationContext automationContext;
    protected String backendURL;

    protected void init(TestUserMode userMode) throws Exception {
        automationContext = new AutomationContext("GREG", userMode);
        backendURL = automationContext.getContextUrls().getBackEndUrl();
    }

    protected void initPublisher(String productGroupName, String instanceName, TestUserMode userMode, String userKey)
            throws XPathExpressionException {
        automationContext = new AutomationContext(productGroupName, instanceName, userMode);
        backendURL = automationContext.getContextUrls().getBackEndUrl();
    }

    protected String getBackendURL() throws XPathExpressionException {
        return automationContext.getContextUrls().getBackEndUrl();
    }

    protected String getSessionCookie() throws Exception {
        LoginLogoutClient loginLogoutClient = new LoginLogoutClient(automationContext);
        return loginLogoutClient.login();
    }

    protected String getServiceURL() throws XPathExpressionException {
        return automationContext.getContextUrls().getServiceUrl();
    }

    protected String getTestArtifactLocation() {
        return FrameworkPathUtil.getSystemResourceLocation();
    }
}

Test case with inherited base class

  
public class SampleDataPopulatorTestCase extends GREGIntegrationBaseTest {

    @BeforeClass(alwaysRun = true, groups = {"wso2.greg"})

    public void initTest() throws Exception {
        super.init(TestUserMode.SUPER_TENANT_ADMIN);
   }

Creating the test case module

When you write test cases that you need to run, first create a sub module under integration. This module should include test case sources and resource files. Following is its structure;

• test-module-1
• src
• test
• java
• resources

Writing test cases

Now its time to write your test cases. As we discussed earlier in the article you can simply derive it from your base class and continue. The pom.xml of your module will be as follows;

  

        org.wso2.governance
        org.wso2.greg.integration.tests
        5.0.0-SNAPSHOT
        ../pom.xml
    

    4.0.0
    Registry Extensibility Test Module
    org.wso2.carbon.registry.test.extensibility
    jar

Your test case module pom.xml should include all the dependencies and plugins that you need to run the tests. There are a few plugins that you should include here. Here is an example pom.xml. Lets look at the plugins that we use here and how to handle dependencies.

Managing dependencies

With products moving to GitHub you need to follow the exact guidelines to use dependencies in test cases. All the dependencies that you need to run the tests should be defined at the top level parent POM file with the version. In the given example dependency below you should note that the scope is defined as test;

  

                org.wso2.carbon.automation
                org.wso2.carbon.automation.engine
                ${test.framework.version}
                test
            

In your test module POM, you should define the dependencies without version and scope. An example is shown below;

  


            org.wso2.carbon.automation
            org.wso2.carbon.automation.engine

In your utility modules, the dependency scope should be overwritten. In the admin-clients module we have defined the dependencies with the scope compile. The reason for overwriting the scope is that we have defined utility modules as java classes compared with test classes in test modules.

  

            org.wso2.carbon.automation
            org.wso2.carbon.automation.engine
            compile
        

You can learn more about dependency scopes here.

Plugins and configurations

Maven Surefire Plugin(maven-surefire-plugin)

This is the plugin that is used to run the test cases. It runs the unit tests of the product. You should note that this plugin generates test reports in plain text and xml formats. To add more usability to the testing by generating reports in HTML format we use Maven Surefire Report Plugin which will be described later.

Under the configurations that belong to this plugin first we have defined the test cases that need to be run. You should note that there are two configuration files stated here;

  

           src/test/resources/testing-server-mgt.xml
          src/test/resources/testing.xml

testing.xml

This describes the test cases that need to run.

testing-server-mgt.xml

This describes any servers or services that you need to start before running the tests.

To run the tests you need to run the carbon server. To do that we have specified the location of product distribution as a property.

  

                            carbon.zip
                            
                                ${basedir}/../../../distribution/target/wso2greg-${project.version}.zip
                            
                        

We define resources location under this plugin as a property.

  

                            framework.resource.location
                            
                                ${basedir}/src/test/resources/
                            
                        

Since we use a wrapped set of listener classes, the default TestNG listener classes should be disabled with the following property;

  

                            usedefaultlisteners
                            false
                        

For code coverage functionality we have defined the following properties as well. The test framework looks for these emma related properties when instrumenting jars and generating coverage reports.

  
${basedir}/target/emma
                        ${basedir}/src/test/resources/instrumentation.txt
                        ${basedir}/src/test/resources/filters.txt
                        ${basedir}/target/emma

Maven Dependency Plugin (maven-dependency-plugin)

This plugin is used to copy and unpack necessary files and artifacts. Following is a code snippet of it;

  

                        copy-emma-dependencies
                        compile
                        
                            copy-dependencies
                        
                        
                            ${project.build.directory}/emma
                            jar
                            emma,org.wso2.carbon.automation.core
                            
                        
                    

Maven JAR Plugin

This plugin is used to build jars.

  

                org.apache.maven.plugins
                maven-jar-plugin
                2.4
                
                    
                        
                            test-jar
                        
                    
                
            

Maven Surefire Report Plugin

This plugin is used to generate html and xml based reports of tests that were run. After you run the test cases, inside the target folder of your test module there will be a folder created for surefire-reports.

  

                org.apache.maven.plugins
                maven-surefire-report-plugin
                2.7.1
                
                    ${basedir}/target/report
                    registry-api-test
                
                
                    
                        integration-test
                        
                            report-only
                        
                    
                
            

Maven Resources Plugin

This plugin is used to copy resources to the output directory. We copy only the test resources here. When you go through execution plans under this plugin, there are several tasks that have been done. The plugin copies resources from the product pack which resides in the local maven repository. You must remember that it doesnt copy resources from the product pack which resides in the distribution folder that you have built.

• copying keystores
• copying axis configs
• copying client modules

Copy resource files

• resources
• artifacts
• GREG
• wsdl
• schema
• policies
• aar
• axis2config
• keystores
• products

  
automation.xml
automationSchema.xsd
emma.properties
instrumentation.txt
filter.txt
testing.xml
testng-server-mgt.xml

Here we will look at must need resources individually and understand what they do.

artifacts.GREG directory

In the artifacts/GREG folder we need to include resource files such as aar, car and wsdl. Test cases will use this resources in run time.

automation.xml

This includes the configurations that we will need to run the test cases. The example provided will have the all the configurations described here. All the elements are explained in the automation.xml. There are some parameters you need to understand. All the other parameters are self explanatory.

The WSO2 Automation Framework provides in-built TestNG listener classes. You need to define which classes you are going to use in order to run the test suites.

  

            
                org.wso2.carbon.integration.common.extensions.carbonserver.CarbonServerExtension
                org.wso2.carbon.integration.common.extensions.usermgt.UserPopulateExtension
            
        

CarbonServerExtension class does the initial configuration background work to run the test cases that are written in the automation framework. This class starts the carbon server on the configured port and does other necessary configurations like jar instrumentation. You can review the source here.

UserPopulateExtension class does the population of users to run the test cases. Note that you need to define those uses in the automation.xml file.

automationSchema.xsd

This is the schema file of automation.xml

emma.properties / filters. txt / instrumentation.txt

These files are used for code coverage. For example, files can be found at the source code location.

testng-server-mgt.xml

This file contains all the in-built TestNG listeners provided by test framework. You can define any other pre-sequences that should be run before the start of the test cases in this file.

testing.xml

This is the configuration file where you need to specify which test cases you need to run and which listener classes to use.

First you need to define listener classes that are used to run the tests.

Then you need to define your test cases. There are two methods to define it;

As a whole package

If you define it as a package, all the test cases in the package will be executed.

As an individual class

This is really useful if you want to identify an issue in failed test cases. Instead of running all the test cases, you can individually run each test case.

Run test cases

mvn clean install : run the test cases

mvn clean install -Dmaven.surefire.debug : debug test case

Summary

To introduce automation to WSO2 products you only need to have a basic knowledge of TestNG. Configuration is all you have to do to run test cases successfully.