How to do Blue-green Deployments and Canary Releases with WSO2 API Manager

1. Introduction

A typical WSO2 API Manager deployment requires an upgrade from time to time. Usually, we need to update the deployment when there’s a fix, patch or critical security upgrade for products used in the solution. When a system upgrade is performed it’s a common requirement to avoid any downtime. Usually, to address this use case, system admins and DevOps follow blue-green, canary deployments to roll out updates. In this article, we will discuss how to do blue-green canary releases in WSO2 API Manager deployment context.

2. WSO2 API Manager WUM Updates and Patch Installation

When we run an API management deployment for a long time, we need to update the deployment when we have a fix, patch or critical security upgrade for products used in the solution. Most of the times these fixes come as a patch or WUM update with only jar or UI level changes. So a majority of these updates don’t have database or schema changes. When we have these kinds of updates or patches we do not need to worry about data replication between the older and the updated deployments. We can keep the data layer as it is and only update the application. To address this use case we will be able to use both blue-green and canary update processes without making the system read-only.

2.1 Canary Release for WSO2 API Manager Product Update and Patch Installation

In general terms, a canary release is a technique used to reduce the risk of switching traffic to a completely new deployment in production by slowly routing a subset of the complete load. Usually, we do this before rolling it out to the entire infrastructure and making it available to everybody. If we consider major WSO2 API Manager versions then we will issue a number of patches and updates for that particular version. Usually, we do not have API level changes or critical non-backward compatible changes on these updates and fixes. So we can easily do canary release when we upgrade the system.

Figure 1: Canary release for WSO2 API Manager

We’ll now explore how we can do a canary release for WSO2 API Manager upgrade. First, we need to take a database dump of the running system. We normally do this as a safety precaution so that if anything goes wrong we can rollback the system to its last working state. When you follow this approach you usually don’t need to block store and publisher operations (even write operations). And we can allow the API gateway (which is more critical for the live system), key manager and traffic manager to serve requests.

To do the canary release you’ll need to follow the steps listed below:

• Create database dump of current running setup (live system). And make sure this backup can be restored without an issue.
• Now it's time to update/patch the lower environment. Then we will need to have deployment similar to production with updated products.
• If it’s a product upgrade or patch update, then we need to follow instructions listed in the readme file. In this case, we also need to set up a complete deployment using the upgraded or patched product distribution. Usually, we use deployment automation tools like Puppet and Chef for this.
• This newly updated deployment should be pointed to the same database system (where you have the working data set). Additionally, deployment directories should be shared across the environment to make sure both deployments have the same data set.
• After performing the upgrade you should do a quick sanity check on the new environment.
• Then you need to test all use cases end-to-end in the updated test deployment.

When you are happy with the new version, you can start routing a fraction of the traffic to this new deployment. There are different strategies to choose when figuring out which traffic should go to the new deployment.

A simple strategy is to send a random set of requests to the newly created deployment. Usually, we can do this by adding a load balancer and a firewall rule saying route x% of traffic to the new deployment. So if we send 10% of traffic to the new deployment and the rest of the 90% will route to the old deployment. Some organizations choose to release the new version to route internal traffic before releasing to the public internet.

In large, distributed scenarios, instead of using a load balancer to decide which requests will be redirected to the new deployment it is also common to use different partitioning strategies. If you have geographically distributed users, you can rollout the updates to one region first and then move to the next regions. If needed, users can have multiple canaries.

One drawback of using canary releases is that you have to manage multiple deployments of WSO2 API Manager at once. You can even decide to have more than two versions running in production at the same time, however, it is best to keep the number of concurrent versions to a minimum.

2.2 Blue-green Deployment for WSO2 API Manager Product Update and Patch Installation

The blue-green deployment also gives us a rapid way to rollback when we need to upgrade or migrate WSO2 API Manager deployments. Here we will move the traffic to the new deployment all at once unlike in canary release mode. If we consider the blue-green deployment, it’s the same basic mechanism as when you need to get a hot standby working. This allows you to test your disaster recovery procedure on every release. The fundamental idea is to have two easily switchable environments.

Figure 2: WSO2 API Manager Blue-green Deployment

Same as the canary mode you need to follow the steps below when doing the real migration on the live system. We need to identify minimum traffic window here too.

• Create a database dump and restore to the new environment.
• Do the migration, system update or patch application in the new environment.
• Then test the functionality using automated and manual tests.
• Once you are confident enough switch the traffic to the new (blue) deployment using a DNS switch or load balancer level change.

3. Migration Between Major Product Versions

If you are doing WSO2 API Manager deployment migration between major product versions, then there will be schema and data changes. In such cases, we normally recommend doing the blue-green deployment. But if we really need gradual traffic movement to the new deployment then we can follow the canary process as well. Before you do any migration you need to analyze traffic patterns of the deployment for about one or more weeks to identify a time slot for the update. System maintainers should identify the minimum traffic window for the update. Once you follow the above steps you can get a rough idea about the time it will take for the final update and you can allocate time slots as per above analysis.

3.1 Blue-green Deployment to Migrate Between Major Product Versions

Usually we take a dump of the current running system and create another new deployment using data from the running system. In WSO2 API Manager’s case, even if we make the store and publisher read-only there can be access tokens generated during the migration process (after we took the dump and before we switch traffic to the new deployment). So, in this case, we need to create a one way sync up script which copies tokens created in the live system to the new system - a script that copies all newly added and updated entries in live system to the migrated/updated system.

The other option is put the complete system to read-only mode before cut-over, run it for a while in read-only mode, and then switch it to read-write mode. But for API Gateway this is not an applicable solution as users shouldn’t see any downtime at all.

3.1.1 How to do blue-green deployment

Now let's see how we can do the blue-green deployment with WSO2 API Manager. Usually, we recommend this pattern when we have to do WSO2 API Manager migration between major versions.

• Create a database dump of the current running setup (live system) and then restore it to the lower environment.
• Do a sanity check. You need to check if you have created the dump and set up the lower environment properly.
• Create and test the one-way sync up script. In blue-green mode, we do not route requests to both environments at the same time. So here we need to think about token tables if we make the store and publisher read-only. Ideally what we need is a way to always copy new entries created in the live deployment to the new deployment.
• Perform migration steps in the lower environment if it is a migration between major versions.
• After performing the migration you should do a quick sanity check.
• Then you need to test all the use cases end-to-end in the migrated test deployment.

We need to follow the steps below when doing a real migration on the live system. Here we need to identify the minimum traffic window too.

• Make WSO2 API Manager store and publisher read-only for the maintenance window.
• Create a database dump and restore to the new environment.
• Do the migration, system update or patch application in the new environment.
• Run the sync up script to update new tokens created in the original deployment.
• Test the functionality using automated and manual tests.
• Once you are confident enough, switch the traffic to new (blue) deployment using a DNS switch or load balancer level change.

Figure 3 illustrates the blue-green model for major product updates.

Figure 3: Blue-green model for major product updates

3.2 Canary Release to Migrate Between Major Product Versions

Similar to a blue-green Deployment, we can start this by deploying the upgraded version of our software into the infrastructure to which no traffic is routed. Then we can start the initial testing on that deployment.

We need to perform the following steps for the final product update:

• Make WSO2 API Manager store and publisher read-only for maintenance window (say 3 hours).
• Create database dump and restore to the new environment.
• Do the system update in the new environment. Usually, this involves data migration and a completely new deployment using the new version of the product.
• Run the sync up script to update new tokens created in the original deployment.
• Test the functionality using automated and manual tests.

When you are happy with the new version, you can start routing a fraction of the traffic to this new deployment. Then you can gradually increase traffic to the new deployment until all traffic is routed.

4. Conclusion

In this article we discussed blue-green and canary release models in the WSO2 context. In WSO2 API Manager deployments, minimum downtime is a very important requirement as many different clients and users invoke APIs (from different geo locations, time zones, etc.) and the API management systems don’t have control over that. So uptime is critical. At the same time it's really important to make sure the system is up to date and critical patches and updates are applied whenever available. When we apply updates to the system we usually need a downtime window. If we follow blue-green and canary deployment patterns we can minimize this downtime. Since these deployment practices are proven patterns we recommend our users to follow these.

Usually when we have very high traffic and we are not sure about completely switching traffic to the new deployment, we can use canary deployment pattern to do gradual traffic movement. This is a good solution for product updates and when patching products (when we have minimum data changes).

If you have fully tested and are confident about the new deployment you may use blue-green pattern. Also we can use blue-green pattern easily if we have a hot backup of the live system. Generally the blue-green pattern is used when we do complete product migration between major versions.