Eagle Technology Group: Applying Agile Software Development Principles to WSO2 Products

Eagle TG is an information technology engineering and integration company based in New Braunfels, Texas. Owned by the Native American Modoc Tribe of Miami, Oklahoma, Eagle TG offers enterprise architecture, virtualization, and data center solutions. They are also a WSO2 Preferred Partner and carry an average of 18+ years of technology expertise per employee.

At WSO2Con USA 2018, Neil Custer, a senior enterprise systems engineer at EagleTG explored their agile process-based configuration management tool for WSO2 products, which they deploy and maintain for one of its largest customers — the headquarters of a branch of the US Department of Defense (DoD). Eagle TG harnessed its considerable in-house software development expertise to develop this new product.

The goal of the new configuration management product is to maintain the consistency of configurable items that affect the performance, functional capabilities and physical attributes of WSO2 products while taking into account operational concerns.

Managing Configurations Before WSO2 Update Manager (WUM)

The release of the WSO2 Update Manager (WUM) in late 2016, had a considerable impact on the way EagleTG did its configuration management. WUM is a simple command line tool that connects to the WSO2 Update service, determines which updates are new and relevant, and downloads them.

Prior to the arrival of WUM, Eagle TG used a painstaking manual process to capture configuration management changes. Using open source tools, EagleTG would evaluate the new out-of-the-box WSO2 product against what they had previously configured. They would then manually update a text file with all of the items that were changing.

The Impact of WUM

So, how did the release of WUM lead to Eagle TG rethinking its configuration management protocols?

WUM meant Eagle TG could no longer use an update patch and simply overlay a couple of files, and perhaps change a parameter or two in a configuration file to enable the product to work in both their lab and in their customer’s environment. With WUM, Eagle TG’s developers needed to wipe out the entire implementation of that product instance with the existing configuration, and do a complete reinstall of the product from scratch. They then had to test the new products in multiple environments.

Doing this with about 6 different WSO2 products was challenging, to say the least. Especially since many of their DoD customers worked in isolated environments where access to the Internet was not a given, which meant that they couldn’t use the Puppet scripts provided by WSO2 to automate the configuration process.

The New and Improved Configuration Management System

Eagle TG’s software and systems engineers realized they would need to rethink their configuration management process if they were to keep providing topnotch services and operations support to their DoD customer. After consulting the team at WSO2 they realized that they needed to script everything they can. The answer lay in harnessing the rich and varied software background of its personnel and recasting configuration management as software development. In other words, Eagle TG’s team decided to treat the items it had to configure in order to deploy a product into specific functions as source-code.

A shared, central repository was created to hold the configurable items for each product. This allowed all team members an efficient way to access these configurable items, modify them, as needed, and to push them back into the repository where others could do the same. All changes are tracked centrally, making it easy for team members to see what has been changed, by whom, when and why. With a simple right click, the repository history is viewable, listing every change that has been made by any team member, along with the date and reason for the change.

Branching while testing is another attractive feature of the new configuration management process. For example, engineers are now able to make a side-line configuration change that might get implemented at a later time, by essentially parking it in the repository in a separate branch. Meanwhile, the main branch stays intact until it is time to merge with the secondary branch.

They were initially going to store the entire zip file of the product home folder in the repository but quickly realized this was not feasible because of the large file size and the multiple products and environments they had to store for. They kept the size of the repository manageable by keeping only the items it has customized in it. A separate folder within the repository for certificate key stores was also set up. Within each key stores folder, every WSO2 product has its own separate environment where the product is deployed.

A cron job was created to ensure updates run smoothly. Set up to run three times a week, this interrogates WUM for any updates to relevant WSO2 products it supports for its customer. If the answer is yes, the updates are automatically pulled to a staging area and an email alert is sent out to everyone on the team. Then the previous fully-configured product instance is backed up. Finally, the new product release is deployed using configuration items and fully tested for the required capabilities.

Because Eagle TG had a strong development background, they were able to follow Agile principles in their configuration as code approach. These include having a shared repository where developers can collaborate, using scripts for “self-installing”, and leveraging Jenkins for continuous integration. They have also adopted continuous delivery and are offering the product as a service on AWS cloud.

To help create their code-based configuration system, they built standardized WSO2 Node “Deployers” for each product in each environment. A deployer is a single consolidated, distributable, compressed file (.zip) that helps deploy the fully-configured updated products. It contains

• The folder structure for enabling the fully-configured product deployment
• A README.txt file outlining the basic environmental requirements for scripts
• Script files for carrying out the product installation
• Configurable items for the product
• WSO2 Product Source

95% of the deployer consists of the WSO2 Product Source, which means they have a low overhead of just 5% that is actually stored in their repository.

The Advantages of Adopting Agile Principles for Product Updates

Although the process of converting their configuration management from manual to automatic through their Agile configuration as code approach has been a long journey, it has proved to be advantageous for Eagle TG in many aspects.

Versioning and traceability of changes: By storing configuration code in a version control system, EagleTG is now able to see who changed what, when and why, which makes it easy to analyze code differences between versions. It is also able to use branches to isolate changes under consideration without affecting the main source code.

Increased security: Running the product as a service allows it to run in its own space with a no log-on user in the Linux environment, making it a lot harder to hack into.

Overall, using software principles in configuration management has allowed EagleTG to streamline processes and take advantage of automation. Each discrete activity is self-documenting, making the process vastly superior to the previously employed manual process that was constantly on the verge of having outdated changes in configuration.

New Businesses Opportunities

This system was among the first to get a fully operational suite migrated to Eagle TG’s DoD client’s managed cloud platform. Eagle TG, together with their client’s engineering team, helped change the DoD branch’s internal processes which made it a lot easier for other programs to migrate to the cloud.

As a technology services provider, Eagle TG’s early success also led to more opportunities such as the one to provide the DoD branch with an updated identity and access management solution. They were also able to show the many advantages of open source software which is yet to be realized in many US Government agencies that opt to go for commercial proprietary software.

Watch Neil's session at WSO2Con USA 2018 here.