pa
2016/02/04
4 Feb, 2016

[Article] Operational Governance & Lifecycle Management of the WSO2 Middleware Platform

  • Nuwan Bandara
  • Director - WSO2
Archived Content
This article is provided for historical perspective only, and may not reflect current conditions. Please refer to relevant product page for more up-to-date product information and resources.

Introduction

Typically, any software project starts with a couple of developers writing some code in their local workspace/sandbox. When such initial proof of concept work is completed the project goes into proper planning, and has a defined timeframe as well as resource allocation. In the initial phases, the environment setup and decision making around these is quite significant. Each project will have a number of environments based on the methodologies and devops policies followed by the organization such as development, testing, quality assurance, staging, and production.

Hence having a governance model to tie up all such moving pieces increasingly seems vital to progress with such a software project. This article tries to summarize just that with a common governance and lifecycle management process for services, applications, and WSO2 middleware artifacts. It will discuss best practices, processors, and standards that can be followed by enterprise application, middleware engineering, and devops groups.


Versioning & source control

Choosing the right version control system often becomes a case of debate; a development team would need to decide between centralized and distributed after evaluating the pros of cons of both of these. Apart from the obvious differences1, the distributed mode is more suitable if the development team is considerably large.

In a large development team, a centralized development mode will result in many source conflicts along the way as complexity grows. Thus, evaluating all of the finer details is crucial in the initial stage. Therefore, picking the right version control system is important as this serves the purpose of storing the source, configuration, and at times binary artifacts.


Continuous integration

From the initial phase itself it's always better to have a continuous build configured with the source repository that eliminates the “works (or builds) in my machine” claim by the developers. Having a central build system and coupling it with the source repository will make it easier to do daily releases of the code and to carry out continuous automated testing2. Continuous integration is essential to release quickly and more often, and enables rapid application development while maintaining the quality.


Versioning and continuous integration for WSO2 middleware artifacts

There are several artifacts that are recommended to be versioned during different project phases. WSO2 middleware products such as WSO2 Data Services Server (WSO2 DSS), WSO2 Enterprise Services Bus (WSO2 ESB), WSO2 Business Process Server (WSO2 BPS), and WSO2 Identity Server (WSO2 IS) generate runtime specific artifacts. These artifacts can be versioned and a build system can generate a deployable archive (Composite Application archive - *.car).

The artifacts can be broadly categorized as follows:

  • Service development - JAX-*, DSS (.dbs), Javascript
  • Application development - .war (spring, jsp), Javascript
  • Integration & choreography - ESB artifacts (sequences, APIs, proxies, endpoints)
  • Orchestration & workflows - BPEL (.bpel), BPMN (.??)
  • API development - APIs (Swagger, API archive)
  • Policy development - XACML, WS-Policy

When these are integrated with a build system (Figure 1) a maven job can create a deployable archive (.car) for these runtime definitions. Refer here3 for more details on how to generate a maven package .pom for any of these artifacts.

Figure 1


Middleware server configuration versioning and build integration

WSO2 products have a couple of system configuration files that are changed commonly based on the use case and deployment (which are, but not limited to, carbon.xml, axis2.xml, catalina.xml, data-sources.xml, registry.xml, jndi.properties, etc.) versioning these configs will also make things easier as that provides the change history and rollback capabilities (these files are stored in the $PRODUCT_HOME/repository/conf directory).

When integrating with the build and deployment, devops need to take a few extra steps to deploy these. A commonly used method is puppet-based deployment.


Environment sensitive configuration

Some of the above artifacts are environment-specific. Endpoints, policies, and configuration definitions like carbon.xml, registry xml are some such artifacts. Versioning based on environment seems more practical in such scenarios.

  • Commonly used environment specific artifacts - ESB endpoints, security and throttling policies, JNDI properties, data sources

Environment separation

Any software project starts with a development phase with a plan to include multiple environments for different functional purposes within an agile process. Based on the organizational policies, there are a standard set of application environments. In a very high level, these environments can be divided into “Development”, “Integration”, “Testing”, “Staging” and “Production”4.

It is important to identify how each of the artifacts behave in these environments and build up the end application at each environment.


Revision control structure and best practices

These are typically the dev boxes with local repositories where developers write the code, experiment, and tweak.

In a small development, if the version control system is subversion this means all developers will commit to a development branch commonly known as the “trunk.”

In a large distributed development team, if the version control system is git (or git flavored) this means the developers will commit to their own local git repositories for development.

When the development is somewhat stable all developers merge their changes so that the integration, testing, and subsequent environments can be created. For subversion, this means creating a stable branch5.

For git, this means merging to a central git repository (Figure 2). More on git-based revision and release management can be found at Gitflow workflow by Vincent Driessen6.

(https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)

Figure 2


Deploying to environments

Once there is a clear branching strategy, continuous integration and deployment are somewhat straightforward. The build system can have a hook to the revision control system (git/svn) and it can trigger a build for a git merge or an svn commit as shown in Figure 3. Depending on the team's behavior this needs to be configured for trigger on each merge or as a scheduled task (a nightly build).

Figure 3

Once the builder job completes its tasks, it will create multiple deployable archives (Figure 4). These can be services archives, application archives, middleware archives, policies and API definitions. Each of these artifacts belongs to a specific deployer so they can participate in the runtime. For this task, “puppet” (or chef) is commonly used7.

Figure 4


Environment-specific configurations

As mentioned earlier, there are certain configurations that are specific to the environment. The JDBC URL is not common across all environments; there is a development database, a separate staging database, and a production database. Similarly, endpoints, policies and certain properties are mostly local to the environment.

In such situations, parameterization becomes a handy tool. The configuration file is simply a template that resides in the revision control system and based on the environment and the puppet takes care of populating the template with environment-specific values (Figure 5).

Figure 5


End-to-end governance - The role of a governance registry

Governance needs to play a major role from the inception of the project. Cataloging the design documents to tracking production, APIs, and processor are part of end-to-end governance.

WSO2 Governance Registry8 can play a major role in connecting these loose ends in a typical software project.


As a dependency management - impact analysis and visualization tool

In any phase of the project, all stakeholders need to know the dependencies and associations of different artifacts. Like the revision control system is the repository for all resources, the governance registry will become the place where the relationships are defined for all resources.

For instance, in the development phase, a team of developers have created a SOAP web service with multiple security policies and schemes and they should be able to communicate this to other teams that wish to use the service. The governance registry can visualize the service and its dependencies describing the details.

Similarly, with impact analysis, developers and stakeholders can get an understanding of the change impact of any of the dependent resources on a particular service, sequence, or an API (Figure 6).

Figure 6

As illustrated in Figure 7, the task of creating assets in the governance registry can be linked to the automated overall process where the builder will create the artifact directly in the governance registry.

Figure 7


As an asset store

In a large enterprise project, there are many types of assets and managing these can sometimes be challenging. The governance registry can functionally catalog these assets, sometimes acting as a static repository and sometimes acting as a dynamic registry.

The repository aspect of the governance registry simply catalogs any document that relates to the project to provide project information and insights.

The registry portion of it can advertise message topics for publish-subscribe MOM, and can act as the API store and the back office application managing API subscriptions and tokens. It can become the policy information point of a policy-based decision system and likewise (Figure 8).

Figure 8


As a life-cycle management tool

Even though different tools participate in the software project’s lifecycle, there is a need for a governed lifecycle management platform. Typically, when the code is branched and the build is running on the staging mode, it’s considered that the lifecycle is now in staging. However, this process doesn’t have any approval processors or an interactive governance model built into it.

The governance registry can tie up these pieces together. Once the active feature development is complete for a particular release and if the integration tests are successful, the project management can trigger an LC promotion of the platform to test, and similarly to staging and ultimately to production. The LC promotion in the governance registry can trigger the corresponding build and subsequently the deployment (Figure 9).

The governance platform can also be used to send notifications to the team on LC state changes and current updates.

Figure 9


Summary

A software project lifecycle is complex and involves many processes; some are standard textbook principles and some are customized per the organization, the project, and team. The tool that’s chosen needs to be flexible enough to accommodate these requirements. There is no silver bullet to nail down the entire span of a software project. For different tasks, a different set of tools will be useful, thus ensuring strong performance and enabling a seamless process.


References

 

About Author

  • Nuwan Bandara
  • Director
  • WSO2