A Story of Two Continuous Delivery Workflows
-
By Asanka Abeyweera
- 21 Dec, 2021
Enterprise Developers
Platform & DevOps Engineers
Photo by Mimi Thian on Unsplash
The Continuous Integration/Continuous Delivery (CI/CD) process is one of the first practices that needs to be implemented when starting a software development project. It is important to get this right while following the continuous delivery principles as it directly affects developer productivity, the quality of the software, and software delivery. A repeatable deployment process is important for continuous delivery and to bring products to the market faster. There is no one-size-fits-all solution when it comes to continuous integration and continuous deployment or continuous delivery.
There are many techniques and automated developer tools to implement a continuous delivery pipeline. It is natural for the development team dynamics and the nature of the software under development to change with time based on user feedback and other suggested improvements. Because of this, the Continuous Integration/Continuous Delivery (CI/CD) process must evolve with time to cater to these changing requirements. The change could be minor or major. This article discusses one such major change done in the Choreo internal continuous integration and continuous delivery workflow with scaling of the development teams and software. Choreo is an internal developer platform designed to accelerate the creation of digital experiences. Build, deploy, monitor, and manage your cloud native applications with ease as you improve developer productivity and focus on innovation.
In this model, the software development practice is thought of as a series of features implemented and deployed to production after meeting quality assurance in different stages of the release process. The important distinction in this continuous delivery approach is that the feature implementation can be kept in a testing environment for a few days before it is tested and promoted to the next environment. Therefore, the new features can be promoted to different environments independently in different time frames.
Figure 1 - Feature branch-based control plane workflow
The deployment process is described using a collection of Kubernetes YAML files since the deployment environment is a Kubernetes cluster. The software is developed and deployed as a collection of microservices. A feature can have changes to one or more services. Assume that a feature is first deployed to the dev environment and then promoted to the staging environment before it is deployed to the production environment. Please note that it is possible to have more intermediate environments in this continuous deployment model. But let’s take just these three environments for simplicity.
Three branches are maintained in the deployment source control repository to match the environments used in the continuous deployment. With this model, whenever a change is pushed to an environment branch, a deployment build is triggered to roll out the new deployment. The challenging part of this continuous delivery model is how we propagate code changes through different environments. The dev branch cannot be merged directly into the staging branch since that will override the staging environment-specific configurations. As a solution to this problem, the main branch was maintained without any environment-specific configurations. Only the commonly shared Kubernetes artifacts were kept in this branch. Feature changes always start from the main branch and are merged into environment branches when ready.
Let’s look at an example to understand this further. A Kubernetes “service” definition file can be easily shared between environments since it typically does not contain any environment-specific information. But a config map will have different values for each environment. Therefore, we can keep the service definition file in the main branch but not the config map definition file. The config map definition file only resides in the environment branches.
The development workflow is defined below in the order the steps are followed. This workflow is summarized in Figure 1.
Features can be held in an environment without blocking other features. Since separate feature branches are maintained, without any additional tooling or technique the features can be tested and promoted independently.
After the Choreo project got some traction and the number of developers contributing to the project increased, managing feature releases, and ensuring continuous delivery became challenging with the existing model. Therefore, the team was looking for a different approach to manage feature releases to make delivery continuous and seamless. The biggest management overhead was due to the usage of separate branches to manage feature releases in the deployment repository. The solution was to batch changes together and promote them over the environments frequently. Instead of keeping code changes in an environment for days, the changes were promoted from environment to environment very quickly after passing all the quality gates. This approach is more towards trunk-based development.
Figure 2 - Branchless deployment repository workflow
Two deployment repositories are used in this model.
It is not possible to design a continuous delivery workflow to fit all the software development projects in the world. Every project’s nature and requirements are different. Therefore, the CD process needs to be evolved over time to cater to these different requirements. It is not possible to pick the best approach out of the explained two models without considering the time and context they are being applied in. Some of the design considerations are the number of components, team size, nature of the deployment resources, the complexity of the system, and time to production.