A Story of Two Continuous Delivery Workflows

Photo by Mimi Thian on Unsplash

The Continuous Integration/Continuous Delivery (CI/CD) process is one of the first practices that needs to be implemented when starting a software development project. It is important to get this right while following the continuous delivery principles as it directly affects developer productivity, the quality of the software, and software delivery. A repeatable deployment process is important for continuous delivery and to bring products to the market faster. There is no one-size-fits-all solution when it comes to continuous integration and continuous deployment or continuous delivery.

There are many techniques and automated developer tools to implement a continuous delivery pipeline. It is natural for the development team dynamics and the nature of the software under development to change with time based on user feedback and other suggested improvements. Because of this, the Continuous Integration/Continuous Delivery (CI/CD) process must evolve with time to cater to these changing requirements. The change could be minor or major. This article discusses one such major change done in the Choreo internal continuous integration and continuous delivery workflow with scaling of the development teams and software. Choreo is an internal developer platform designed to accelerate the creation of digital experiences. Build, deploy, monitor, and manage your cloud native applications with ease as you improve developer productivity and focus on innovation.

Feature branch-based deployment workflow

In this model, the software development practice is thought of as a series of features implemented and deployed to production after meeting quality assurance in different stages of the release process. The important distinction in this continuous delivery approach is that the feature implementation can be kept in a testing environment for a few days before it is tested and promoted to the next environment. Therefore, the new features can be promoted to different environments independently in different time frames.

Figure 1 - Feature branch-based control plane workflow

Deployment repository structure

The deployment process is described using a collection of Kubernetes YAML files since the deployment environment is a Kubernetes cluster. The software is developed and deployed as a collection of microservices. A feature can have changes to one or more services. Assume that a feature is first deployed to the dev environment and then promoted to the staging environment before it is deployed to the production environment. Please note that it is possible to have more intermediate environments in this continuous deployment model. But let’s take just these three environments for simplicity.

Three branches are maintained in the deployment source control repository to match the environments used in the continuous deployment. With this model, whenever a change is pushed to an environment branch, a deployment build is triggered to roll out the new deployment. The challenging part of this continuous delivery model is how we propagate code changes through different environments. The dev branch cannot be merged directly into the staging branch since that will override the staging environment-specific configurations. As a solution to this problem, the main branch was maintained without any environment-specific configurations. Only the commonly shared Kubernetes artifacts were kept in this branch. Feature changes always start from the main branch and are merged into environment branches when ready.

Let’s look at an example to understand this further. A Kubernetes “service” definition file can be easily shared between environments since it typically does not contain any environment-specific information. But a config map will have different values for each environment. Therefore, we can keep the service definition file in the main branch but not the config map definition file. The config map definition file only resides in the environment branches.

Developer workflow for feature propagation

The development workflow is defined below in the order the steps are followed. This workflow is summarized in Figure 1.

1. As described in the previous section, feature changes are done by taking the main branch as the base branch. Therefore, the first step is to create the base-feature branch using the main branch.
2. Create a PR to the main branch with non-environment-specific changes. This change can be merged to the main branch after review.
3. Create the dev-feature branch by taking the dev branch as the base branch. Then the base-feature branch is merged into the dev-feature branch. If there are any environment-specific changes related to the feature, these changes are then done on the dev-feature branch and a PR is created against the dev branch.
4. The changes are deployed to the dev environment after the changes are reviewed and merged into the dev branch.
5. Execute all the automated and manual tests in the environment related to the feature. If defects are identified related to the feature,
1. Do the changes in the base-feature branch and merge it to the main branch
2. Merge the base-feature branch to the dev-feature branch.
3. Apply any environment-specific changes to the dev-feature branch.
4. Review and merge changes in the dev-feature branch to the dev branch.
6. Repeat steps 3-5 for stage and prod until the feature is completed and delivered.

Advantages

Features can be held in an environment without blocking other features. Since separate feature branches are maintained, without any additional tooling or technique the features can be tested and promoted independently.

Disadvantages and risks

1. Risk due to environment PRs being created separately. There are some scenarios where the environment PRs are different, as a config addition. But if we take a component version update, prod PR being different to the stage can lead to unexpected errors.
2. Feature dependency is hard to detect automatically. Let’s say there are two interdependent feature PRs merged to the staging together. Since the stage changes can be batched the automation test can pass when the dependent changes are applied together. If we move one of these PRs to prod without the other dependent PR, there can be some sort of failure in the production. This is a hard-to-manage overhead from the SRE point of view.
3. Hard to detect when base changes are done in env feature branches directly. Ideally, the base changes should be done in a feature base branch and added to the feature env branches via a merge. By mistake, developers can do changes to base directly in the environment PRs. This can lead to inconsistencies between environments.
4. Delaying the progress of features can lead to many conflicts when merging the feature PRs. This model allows developers to keep a change in a lower environment for a very long time progressing to prod. The result of this is the main branch having a lot of unreleased feature changes. This can result in complex merge conflict resolving scenarios.

Branchless deployment workflow

After the Choreo project got some traction and the number of developers contributing to the project increased, managing feature releases, and ensuring continuous delivery became challenging with the existing model. Therefore, the team was looking for a different approach to manage feature releases to make delivery continuous and seamless. The biggest management overhead was due to the usage of separate branches to manage feature releases in the deployment repository. The solution was to batch changes together and promote them over the environments frequently. Instead of keeping code changes in an environment for days, the changes were promoted from environment to environment very quickly after passing all the quality gates. This approach is more towards trunk-based development.

Figure 2 - Branchless deployment repository workflow

Deployment repository structure

Two deployment repositories are used in this model.

• The common deployment repository hosts all non-environment-specific Kubernetes object configurations.
• The deployment environment overlay repository hosts all environment-specific overlays. These configs are added on top of a common deployment version to get the Kubernetes objects for an environment. Separate branches will be used in the repository to maintain environment-specific overlays separately

Developer workflow for feature propagation

1. Add all the environment-specific configurations to the deployment environment overlay repository. It is important to do this before adding the non-environment specific configs for the changeset propagation through environments to be smooth
2. Add the non-environment specific configs to the common deployment repository
3. Changes will be automatically propagated through the lower environments and then to the production environment. In the case of a failure, corresponding developers will be alerted.

Advantages

• Change dependencies are automatically managed. In this model, the changes are batched and propagated through the environments. Therefore, the process ensures that the dependent changes are pushed together.
• Less room for human errors since the deployment is versioned and propagated through the environment. In the previous model, the environment PRs were created using the base-feature branch. There was some risk involved in this since this task was done by the developer.

Disadvantages

• A change cannot be held in a lower environment for days. This will block other changes from being propagated to production. Therefore, when a change is causing problems in an environment, it needs to be reverted or fixed immediately. This can be overcome by using a mechanism, like feature flags, in some scenarios.
• Since the changes are batched and propagated, small changes can take longer than the previous model to reach production.

Closing thoughts

It is not possible to design a continuous delivery workflow to fit all the software development projects in the world. Every project’s nature and requirements are different. Therefore, the CD process needs to be evolved over time to cater to these different requirements. It is not possible to pick the best approach out of the explained two models without considering the time and context they are being applied in. Some of the design considerations are the number of components, team size, nature of the deployment resources, the complexity of the system, and time to production.