Table of contents

• Introduction
• Introduction to automation on WSO2 platform
• Testable modules in WSO2 platform
• Unit test
• Integration test
• Platform test
• Cross platform testing
• Performance and security scenarios
• Why platform wide test automation?
• Why a single framework?
• Architecture and organization of Test Automation Framework
• Technology overview
• WSO2 Test Automation Framework engine
• WSO2 Test Automation Framework extensions
• WSO2 Test Automation Framework API
• WSO2 Test Automation Framework utils
• Conclusion
• References

Introduction

WSO2 is a completely open-source SOA middleware company that offers a complete SOA enterprise solution from ground up through its award-winning Carbon platform. All WSO2 products are mainly based on the Carbon platform and shares features of the OSGI framework. All products also support both server-based deployments and cloud-based deployments based on Apache Stratos. Moreover, all products have the capability to operate as a simple user in multi-tenant environment.

In order to address the complexity of the platform, test automation in WSO2 goes far beyond the traditional automation mindset of automating testable scenarios. Therefore, we needed a single framework to handle all so-called complexity and perform maximum coverage with the limited number of test cases we have.

WSO2 Test Automation Framework enables performance as a automation framework addressing the absence of a single framework that can perform equally in all stages of the development and deployment cycles.

Meanwhile, the targeted user group would be the developers themselves. Therefore, the main target is to provide an easy-to-use framework to automate, encapsulating all complexities with automating the particular functionality.

Introduction to automation on WSO2 platform

The WSO2 platform evolves at a rapid pace, so the introduction of new features and addition of hundreds of lines of codes are not a rare occurrence in day-to-day operations. Moreover, to preserve the traditional nature of open-source software, WSO2 tends to go for rapid releases. Hence, there is a significant requirement to move towards a stable, continuous integration and delivery platform. As a result, automated tests plays a major role in delivery and support areas. To this end, WSO2 needs a stabilized automation platform to achieve this and a generalized mechanism to monitor progress.

WSO2 uses OSGI-based carbon as a single platform for all its products. Carbon is the main foundation as all other features are plugged as OSGI bundles. So, the boundary of the product is determined by the set of common features bundled to serve a specific purpose. However, a key feature of the WSO2 platform is that several sets of features can be added to a product that can extend the product’s capabilities, e.g. we can install data service features into the Application Server to make the WSO2 Application Server work as a Data Service server.

Testable modules in WSO2 platform

In terms of WSO2 package structure, Kernel is the core module that works as a hub; all other modules will be plugged into the Kernel. The WSO2 platform contains many integration patterns that provide security, reliability, and performance. Therefore, the testing pyramid should be modified and some cross-boundary testing modules should be introduced to the equation.



Figure 1: Test organization

As illustrated in Figure 1, we discuss three types of tests in the WSO2 platform that cover the overall picture of automation in the platform.

Unit test

In any platform, unit tests plays a vital role. In WSO2, each OSGI component that is meant to be bundled as a product starts its life as a feature. So, unit tests are vital in order to verify the functionality of the individual feature. Moreover, there are some vital components that will be used throughout the platform like ‘user management’ and ‘authentication features’ that need individual attention to ensure they are working as expected. For unit tests, the black box approach is used. In WSO2, most of the key features and the Carbon kernel itself are equipped with unit tests to verify functionality.

Integration test

Integration tests are the most focused areas in WSO2 automation. Each product hosts an integration test module that is expected to cover all testable scenarios within that product scope. In Integration testing, the product is considered as a single unit and we are writing tests that are only specific for that product within the integration test module.

For integration tests, we are mainly focusing on back-end tests, which mainly tests the backend of the product using Admin services. For other scenarios, which needs to touch the UI interface, we use a selenium and page object model-based approach to design UI integration tests.

Platform test

Platform tests cover the test cases that test integration of several products as a platform. The mostly covered areas under platform tests are

1. Testing clustered scenarios
2. Cloud-based deployments and scenarios based on multi tenancy
3. Security-based scenarios using WSO2 Identity Server
4. Deployment synchronization scenarios

Cross platform testing

Basically, cross platform tests are tests that feature installation and functionality of those features. The WSO2 platform is powered by WSO2 Carbon, which is a OSGI container. This facilitates features to be installed as OSGI bundles. Cross platform testing is not in line with integration tests. Cross platform tests verify a feature that will be installed in multiple products and verifies its functionality.

Why platform wide test automation?

As the code base grows and the technological areas that are covered widens, the traditional approaches of reaching quality goals have become obsolete. Companies like WSO2 have an aggressive delivery practice and aggressive research-oriented operation; these help to minimize the cost per QA cycle due to a steady automation approach.

In WSO2, we follow several integration and deployment patterns and products are used with a mix of third-party platforms, such as Oracle and Microsoft. Therefore, the main challenge is to keep each integration point steady in each release and verify the functional accuracy in each release with minimal cost. In order to achieve these requirements, WSO2 needs a mechanism other than testing single units or an integrated single product.

Other reasons why we need a platform-wide automation is that we need to simulate user stories and make sure a user who tries a deployment with a deployment pattern we have recommended will not have a bad experience. Hence, we need to automate integration patterns, customer deployments, and QSO-based scenarios in order to verify that all integrations are up to a healthy level with each release.

Why a single framework?

The reason for using a single framework and making ourselves complex has been argued; however, our response is that in the end, it’s the developers who write automated tests, so it’s convenient for them to deal with a single framework in each scenario.

Another factor is that there should be a unified way to present coverage details. Code coverage is the main metrics we use to illustrate the progress of the automation. In the end, we are presenting cumulative coverage details, including unit, integration, and platform tests.

Another key reason is that we need to reduce the effort in writing tests and avoid duplication as much as possible. Therefore, the automation framework supports the execution of a single test in each of the mentioned testing scenarios without manual intervention. Since it is a single framework underneath, there should be no changes to the test, so it will execute out of the box in any given execution.

Architecture and organization of WSO2 Test Automation Framework

Writing an automation test from scratch is a complex and time-consuming task. You have to consider many things like server management, environment management, artifact management, reporting, etc. The WSO2 Test Automation Framework takes care of the complicated processes. It provides facilities to

1. Manage several products inside a single test
2. Manage users in several product domains
3. Deploy artifacts for different products under different users
4. Run same test in both local and cloud environments
5. Report TestNg and surefire reports for all your tests
6. Provide tooling support of Selenium, Jmeter, and Soap UI Without having these tools in your local machine
7. Provide automatic authentication
8. Configure admin services as test-oriented API
9. Retrieve environment variables easily
10. Provides a facility to run any third-party products as pluggable modules

Figure 2: Test organization - WSO2 platform

As shown in Figure 2, the WSO2 Test Automation Framework based test consists of several building blocks.

1. Test automation framework
1. Automation engine
2. Automation pluggable extension
2. Support modules
1. Admin service API
2. Reusable API modules
3. Test
4. Configuration

Technology overview

The WSO2 Test Automation Framework is mainly based on the TestNG[1]. the TestNg works as the main test engine inside the framework. The WSO2 Test Automation Framework heavily uses the listener-based test execution of the TestNG. Apart from this we use Maven [2] as the main build technology and use SureFire plugin[3] to manage test execution and report generation.

In automation, coverage is considered as one of the key measuring factors. we use the MOJO approach of EMMA to generate coverage reports for all integration tests.

The WSO2 Test Automation Framework provides seamless support for well-known automation tools like JMeter. Apart from middleware backend automation, the WSO2 Test Automation Framework provides a UI automation facility through Selenium.

Figure 3: Organization - test automation framework modules

The WSO2 Test Automation Framework is mainly based on TestNG[1]. TestNg works as the main test engine inside the framework.

WSO2 Test Automation Framework engine

The test automation engine is the main executable engine of the framework. It is a lightweight and easy to adopt, simple core. The core includes all core features of the framework.

Figure 4: Organization - test framework engine

WSO2 Test Automation Framework extensions

The WSO2 automation framework extensions provide the flexibility to run any Java-based client within TestNg listeners. You can add any custom servers like Tomcat, Apache FTP, Activemq addons tooling support like, Jmeter or any tools and utility modules to be executed with the framework.

WSO2 Test Automation Framework API

Considering backend automation of WSO2 middleware products, automation mostly depends on the Admin services. However, in order to write a well-managed test case, developers need to have a more testing friendly set of APIs for better performance of the automated test case.

Therefore, the API module is basically a testing-friendly wrapping of the Service stubs of WSO2 Admin Services. The API also includes

• Test-friendly client API wrapping service stubs with a unified approach to perform, verify, and assert each admin service
• Testing friendly functions for verification and assertion of functions
• Updates with the relevant release version
• Encapsulates the complexity of changing all available tests in a case of a stub change
• Maintain page object classes for Selenium automation

WSO2 Test Automation Framework utils

When writing a test case, the developers need frequently-used utility components to achieve the functionality of the automated test. Most times the needs are the same like sending SOAP or REST requests to a known endpoint or monitoring a port that is acquired or free.

Some sample utilities would be

• Axis2 client
• Wire message monitor
• Custom server startup scenarios (Axis2, Tomcat, ActiveMQ)
• Concurrency test scenarios

Therefore, to handle such duplication requirements, the WSO2 Test Automation Framework provides a common set of supporting utility classes. By adding the Utils module to the test case, a developer can easily access the utility classes within the test case.

Conclusion

Enterprises today need to deliver high quality software within a short time period and with minimal effort. Test automation is one of the core means to increase your velocity, test efficiency, and enable faster delivery. WSO2’s way of agile practice relies on faster feedback, which is achieved by testing each change incrementally. Robust automation solutions can relieve you of many challenges in software development.

References

1. “TestNG - Welcome.” [Online]. Available: https://testng.org/doc/index.html [Accessed: 07-Feb-2014]
2. “Maven - Welcome to Apache Maven.” [Online]. Available: https://maven.apache.org/ [Accessed: 07-Feb-2014]
3. “Maven Surefire Plugin - Introduction.” [Online]. Available: https://maven.apache.org/surefire/maven-surefire-plugin/ [Accessed: 07-Feb-2014]