[Article] Multi-Environment Artifact Management for WSO2 Products Using WSO2 Governance Registry
By Nadeesha Gamage
- 13 Jul, 2015
In a typical organization, the development team would work on the development environment where all the code is developed, and the QA team would work in the QA environment where they will test the work that is done by the developers. Once everything is tested and the QA sign-off is given, the code is deployed to the production environment. These environments are managed in isolation by a devops team who ensures that the right content is migrated from one environment to another at the correct time.
Even though the above-mentioned practice is common in software development, it has equal applicability when developing integration solutions. Artifacts need to be developed, tested and put to production in isolation by the devops team. However, some artifacts would have environment specific values that make it difficult to simply migrate them from one environment to another. This article looks at how WSO2 Governance Registry can be used to mitigate this problem and how it can manage artifacts of WSO2 products in a multi environment deployment.
Artifact development, bundling, and deployment
WSO2 Developer Studio is an Eclipse-based IDE that provides a developer an environment through which artifacts relating to WSO2 products can be developed. Artifacts developed from the developer studio can be bundled in a Carbon Archive (CAR) file which can be deployed directly to WSO2 servers. The artifacts that need to be tested can be bundled inside a CAR file shared with the QA team for testing. If the QA team signs off the release, the CAR will be deployed to the production environment. This method ensures a reliable way of artifact migration and deployment from one environment to another.
Even though CAR-based artifact migration allows artifacts to be bundled to a single file and migrated to multiple environments, it may not be a solution to all problems faced by the devops teams. In an ideal migration scenario, the artifacts that are migrated would remain the same irrespective of the environment to which it is deployed. This ensures that what is developed is correctly tested and applied in the production environment. Yet, in reality, this doesn’t happen as expected. Most artifacts would contain values that are specific to an environment, and in most cases, these environment-specific values are stored in a configuration file and referred to in each environment. However, as the complexity increases, maintaining such a file becomes complex and erroneous. The architecture of WSO2 products provides an alternative to this where it allows all environment specific values, such as endpoint URLs, to be stored in a registry and refer to them as registry keys from WSO2 products.
WSO2 Developer Studio for artifact bundling
Artifact bundling is the first step in the artifact migration process. WSO2 Developer Studio can be used to create and bundle artifacts that can be deployed to WSO2 products. Product artifacts relating to products, such as the ESB, data services server, governance registry, and business process server can be developed using WSO2 Developer Studio. Once these artifacts are developed, they can be bundled to a CAR file and deployed in the WSO2 server. Artifacts belonging to multiple products can be bundled in a single CAR file and deployed. Each WSO2 server will extract and deploy the artifacts relating to that particular product. This has been illustrated in Figure 1.
Carbon archive files for artifact distribution
A carbon archive (CAR) file can be used as an artifact distribution mechanism whereby the required project artifacts can be bundled in this CAR file and distributed to different environments in the deployment. The devops team only needs to deploy this CAR to WSO2 products in each environment, and each product will deploy the artifacts that are relevant to that particular product.
Even though this method allows artifacts to be deployed across environments, artifacts itself would have references to each individual local environment. For example, an ESB sequence would refer to an endpoint, which will be a mock service endpoint in both the Dev and QA environment, but the same endpoint should refer to the actual service in the Production environment. This can be facilitated by defining these endpoints in a central registry as opposed to defining them in the ESB artifact itself. ESB artifacts would simply refer to a registry location that would have the endpoint information relating to the given environment.
WSO2 Governance Registry
WSO2 Governance Registry is a registry and a repository to store any type of data or metadata. It provides a rich set of features including SOA governance, lifecycle management, and a strong framework for governing anything. In this case, WSO2 Governance Registry would be used to manage registry artifacts that are stored in a common registry space. WSO2 Developer Studio can be used to create registry artifacts that can be deployed to WSO2 Governance Registry. Artifacts, such as endpoints, can be created and bundled from WSO2 Developer Studio and deployed to a WSO2 Governance Registry instance. WSO2 Governance Registry would share these registry artifacts with other WSO2 products in deployment. This can be depicted by the diagram in Figure 2.
To explain this concept we will assume that there are 2 environments (development and production) and an ESB proxy service would need to be deployed to each of these environments. The proxy service should call different backend services based on the environment to which it is deployed. Each ESB has its registry space mounted to a shared registry, which will hold the artifacts that are specific to the environment.
WSO2 products have 3 major partitions in the registry, a local space, a configuration space, and a governance space.
Local registry space - Local registry space is specific to each WSO2 product and is not shared with other products in the deployment. Local registry stores artifacts that are local to that particular server.
Config registry space – Config registry is shared between similar products in the cluster. For example all ESB nodes in the cluster would share a single Config registry space. All product specific artifacts are stored inside this registry.
Governance registry space – This registry space should not be confused with WSO2 Governance Registry. The governance registry space is a common space that is shared by all WSO2 products in the deployment. This registry space will store all artifacts that are common to all WSO2 products in the deployment.
Registry mounting is a way by which WSO2 products link its config and governance registry spaces to a shared registry. This allows artifacts that are included in the config space to be shared among similar WSO2 products in the cluster, and to share artifacts in the governance space with all WSO2 products in deployment.
In a multi-environment deployment as the one that is explained here, each environment would have its own config and governance space. You can manage artifacts in these registry spaces using a common WSO2 Governance Registry product instance. Keeping all these points in mind, we can illustrate the registry relationship in Figure 3.
Config registry space in the both the development and production environments would point to ‘/_system/conf/ESB/Dev’ and ‘/_system/conf/ESB/Prod’ respectively in the shared registry. All the artifacts that are common to all ESB instances would reside inside these spaces. Ideally both these spaces should have content with identical names, but the configuration inside these artifact files should have different values.
Governance Space of the ESB instances in development and production environments would point to ‘/_system/governance/Dev’ and ‘/_system/governance/Prod’ respectively in the shared registry. As you would have noticed the governance space doesn’t contain any product specific folder structure as the governance space is shared by all the products in the deployment. Product artifacts that are common to all the products would reside in this registry space.
WSO2 Governance Registry is also mounted to the shared registry and would have visibility and control of artifacts on all environments. WSO2 Governance Registry would be the central point from which artifacts are pushed and managed in the shared registry.
Making it all work together
Figure 5 illustrates the registry mounting scenario.
Let’s start with the Governance Registry. WSO2 Governance Registry manages artifacts that are shared in the common registry spaces. These include both the config and governance spaces in the registry. Registry artifacts that are created from the developer studio can be deployed to the shared registry through WSO2 Governance Registry. Each environment would have its own folder in both config and governance registry spaces. If you refer to the diagram in Figure 5, endpoint Ep1 would be in both dev and prod folders in the governance space. Even though the endpoint names are the same in both environments, the endpoint would refer to two different backend services.
The ESB artifact would contain proxy services, sequence, and other ESB configuration that would simulate a message flow. ESB artifacts would refer to endpoints using a registry key rather than pointing to actual endpoint URL. The registry key is a reference point in the registry where the endpoint details are stored. As described in the diagram above a registry endpoint (within the governance space) in the ESB can be referred as ‘<address uri=/_system/governance/Endpoints/Ep1.xml>’. As you would have noticed, the reference is the same in both dev and prod environments, which means that the same CAR file containing the ESB artifacts can be deployed to both environments.
ESBs in both environments are mounted into a shared registry. However, they would mount to different folders in the governance space. The ESB instance in the development environment mounts to a ‘/_system/governance/dev/..’ whereas the ESB in the production environment mounts to ‘/_system/governance/prod/..’. Hence, the endpoint reference ‘/_system/governance/Endpoints/Ep1.xml’ would refer to different endpoint URL’s based on the environment in which the configuration is deployed. Even though this use case refers to artifacts in the governance space, similar mounting mechanism is followed for artifacts in the config space.
WSO2 Governance Registry can support management of artifacts relating to WSO2 products in a multi-environment deployment. Moreover, artifacts relating to WSO2 products can be bundled and deployed to different WSO2 product instances and you can isolate product and environment specific registry artifacts. This type of artifact deployment would allow devops teams in the organization to effectively manage the migration of artifacts between different environments with minimum effort.
- Nadeesha Gamage
- Associate Technical Lead