Tag Archives: Products

Implementing an Effective Deployment Process for WSO2 Middleware


Image reference: https://www.pexels.com/photo/aerospace-engineering-exploration-launch-34521/

At WSO2, we provide middleware solutions for Integration, API Management, Identity Management, IoT and Analytics. Running our products on a local machine is quite straightforward: one just needs to install Java, download the required WSO2 distribution, extract the zip file and run the executable.

This provides a middleware testbed for the user in no time. If the solution needs multiple WSO2 products, those can be run on the same machine by changing the port-offsets and configuring the integrations accordingly.

This works very well for trying out product features and implementing quick PoCs. However, once the preliminary implementation of the project is done, a proper deployment process is needed for moving the system to production. 

Any software project needs at least three environments for managing development, testing, and the live deployments. More importantly, a software governance model would be needed for delivering new features, improvement, bug fixes and managing the overall development process.

This becomes crucial when the project implements the system on top of a middleware solution. Both middleware and application changes will need to be delivered. There might be considerable amounts of prerequisites, artifacts and configurations. Without having a well-defined process, it would be difficult to manage such projects efficiently.

A High-Level Examination

One would have to consider the following points would need to be considered when implementing an effective deployment process:

  • Infrastructure

WSO2 middleware can be deployed on physical machines, virtual machines and on containers. Up to now most deployments have been done on virtual machines.

Around 2015, WSO2 users started moving towards container-based deployments using Docker, Kubernetes and Mesos DC/OS. As containers do not need a dedicated operating system instance, this cuts down resource requirements for running an application – in contrast to a VM. In addition, the container ecosystem makes the deployment process much easier using lightweight container images and container image registries.

We provide Puppet Modules, Dockerfiles, Docker Compose, Kubernetes and Mesos (DC/OS) artifacts for automating such deployments.

  • Configuration Management

The configuration for any WSO2 product can be found inside the relevant repository/conf folder. This folder contains a collection of configuration files corresponding to the features that the product provides.

The simplest solution is to maintain these files in a version control system (VCS) such as Git. If the deployment has multiple environments and a collection of products, it might be better to consider using a configuration management system such as Ansible, Puppet, Chef or Salt Stack for reducing configuration value duplication.

We ship Puppet modules for all WSO2 products for this purpose.

  • Extension Management

WSO2 middleware provides extension points in all WSO2 products for plugging in required features.

For example, in WSO2 Identity Server a custom user store manager can be implemented for connecting to external user stores. In the WSO2 Integration products, handlers or class mediators can be implemented for executing custom mediation logic.  Almost all of these extensions are written in Java and deployed as JAR files. These files will simply need to be copied to the repository/components/lib folder or the repository/components/dropins folder if they are OSGi compliant.

  • Deployable Artifact Management

Artifacts that can be deployed in repository/deployment/server folder fall under this category. For, example, in the ESB, proxy services, REST APIs, inbound endpoints, sequences, security policies can be deployed in runtime via the above folder.

We recommend that you create these artifacts in WSO2 Developer Studio (DevStudio) and package them into Carbon Archive (CAR) files for deploying them as collections. WSO2 DevStudio provides a collection of project templates for managing deployable files of all WSO2 products. These files can be effectively maintained using a VCS.

These files can be effectively maintained using a Version Control System.

  • Applying Patches/Updates

Patches were applied to a WSO2 product by copying the patch<number> folder which is found inside the patch zip file to the repository/deployment/patches/ folder.

We recently introduced a new way of applying patches for WSO2 products with WSO2 Update Manager (WUM). The main difference of updates, in contrast to the previous patch model, is that fixes/improvements cannot be applied selectively; it applies all the fixes issued up to a given point using a CLI. This is the main intention of this approach.

  • Lifecycle Management

In any software project it is important to have at least three environments – one for managing development, one for testing and one for production deployments.  New features, bug fixes or improvements need to be first done in the development environment and then moved to the testing environment for verification. Once the functionality and performance are verified the changes can be applied in production (as explained in the “Rolling Out Changes”) section.

The performance verification step might need to have resources identical to the production environment for executing load tests. This is vital for deployments where performance is critical.

With our products, changes can be moved from one environment to the other as a delivery.  Deliveries can be numbered and managed via tags in Git.

The key advantage of using this approach is the ability to track, apply and roll back updates at any given time.

  • Rolling Out Changes

Changes to the existing solution can be rolled out in two main methods:

1. Incremental Deployment (also known as Canary Release).

The idea of this approach is to incrementally apply changes to the existing solution without having to completely switch the entire deployment to the new solution version. This gives the ability to verify the delivery in the production environment using a small portion of the users before propagating it to everyone.

2. Blue-Green Deployment

In Blue-Green deployment method, the deployment is switched to the newer version of the solution at once. It would need an identical set of resources for running the newer version of the solution in parallel to the existing deployment until the newer version is verified. In case of failure, the system can be switched back to the previous version via the router. Taking such approach might need a far more thorough testing procedure compared to the first approach.

Deployment Process Approach 1

This illustrates the simplest form of executing a WSO2 deployment effectively.

In this model the configuration files, deployable artifacts and extension source code are maintained in a version control system. WSO2 product distributions are maintained separately in a file server. Patches/updates are directly applied to the product distributions and new distributions are created. The separation of distributions and artifacts allows product distributions to be updated without losing any project content.

As shown by the green box in the middle, a deployable product distribution is created, combining the latest product distributions, configuration files, deployable artifacts and extensions. Deployable distributions can be extracted on physical, virtual machines or containers and run. Depending on the selected deployment pattern, multiple deployable distributions will need to be created for a product.

In a containerized deployment, each deployable product distribution will have a container image. Depending on the containerized platform, a set of orchestration and load balancing artifacts might also be used.

Deployment Process Approach 2

In the second approach, a configuration management system has been used for reducing the duplication of the configuration data and automating the installation process.

Similar to the first approach, deployable artifacts, configuration data and extension source code are managed in a version control system. Configuration data needs to be stored in a format that is supported by the configuration management system.

For an example, in a Puppet configuration, data is either stored in manifest files or Hiera YAML files. Deployable WSO2 product distributions are not created. Rather, that process is executed by the configuration management system inside a physical machine, virtual machine or in a container at the container build time.

In conclusion

Any of the deployment approaches we’ve spoken about above can be followed with any infrastructure. If a configuration management system is used, it can be used for installing and configuring the solution on virtual machines and as well as on containers. The main difference with containers is that configuration management agent will only be triggered at the container image build time. It may not be run in the when the container is running.

If a configuration management system is used, it can be used for installing and configuring the solution on virtual machines and as well as on containers. The main difference with containers is that configuration management agent will only be triggered at the container image build time. It may not be run in the when the container is running.

At the end of the day, a proper deployment process is essential. For more information and support, please reach out to us. We’d be happy to help.

How we handle security at WSO2

A Proactive Strategy for Security Management

Any decent software development organization generally has a well-defined set of policies and procedures for security management.

At WSO2, we – as in, the Platform Security Team – constantly collaborate with other product teams, customers and external security researchers to manage overall security of all WSO2 product. In this post, we’d like to talk about how we do this.


Part One: in the realm of code

code-944504_1280

I: Designing for security

The first stage of software design is the gathering of requirements. In open source software, we tend to use third-party code quite a bit – it’s how open source works: we stand on the shoulders of giants.
However, we can’t simply use what code we think is suitable.

The first check comes here. At WSO2, if we identify any kind of third-party code to be used, we need it to be first approved by the Engineering Management group, who are an internal group of seasoned architects who function at a directorial level. For us, security comes as a first priority, not as an afterthought.

The next set of checks come in the design phase. What are the communication protocols being used? How secure are they? Where is the data stored, and how? What endpoints are we exposing to the public? We go through a series of use cases to identify where this design can be broken, and work with the product design team to integrate our security concerns from the start.

II: Review, rinse, repeat

The next part is obvious: every developer is responsible for writing clean code [1, 2, 3].

Code written by each developer goes through a process of code quality reviewing overseen by members of the relevant product team and the Platform Security Team. When submitting the code for reviewing, the developer has to submit the static code analysis reports – generated using tools like FindSecBugs [4]. This is a mandatory security check in the reviewing process. Only upon fixing all issues spotted in the first pass is code is merged to the repository.

III: Testing with the automated grindhouse

At WSO2, we use Jenkins quite a lot for automating the build process. It builds individual components; it packages components together; it constantly builds and re-builds.

A large part of our security testing is integrated right into this process. Jenkins first performs the OWASP Dependency Check [5, 6], which analyzes the project dependencies and produces vulnerability reports. Even after the selection process in the first stage is complete, there can be some vulnerabilities that we haven’t spotted – especially if they’ve only been discovered extremely recently.

Next, Jenkins uses FindSecBugs as a plugin; during each automated build cycle, it checks individual components and generates vulnerability reports, which are in turn submitted to the security team for review.

Jenkins also uses the OWASP Zed Attack Proxy for dynamic code analysis [7, 8]. During the dynamic security analysis, the entire URL tree of the product is scanned and well-known attacks and exploits are automatically performed; the results are reported. These reports, too, are investigated by the respective product team as well as the Platform Security Team.

Once the testing is complete and a product is ready to be released, the respective product team has to receive security clearance from the Platform Security Team. If any known vulnerabilities are still listed in the reports, the product team has to justify to us the existence of the reported vulnerability – a pretty hard job.

We find that developers may write code following all the best security practices, but when the code is merged together, it might still open up a vulnerability because of how everything integrates together.


 Part Two: when humans happen

astronaut-and-robonaut-shake-hands

I: Preparing for the real world

There’s a saying: no battle plan survives contact with the customer. Although security standards and processes are followed to the letter, our products have to run in the real world.

One of the most important things is building awareness. We put together a set of deployment patterns, security recommendations, and best practices to be followed when deploying our products; we also conduct public webinars for making awareness in security related topics for WSO2 users, which are available at wso2.com/library/webinars.

II: Building internal Champions

Sometimes there is a gap between the product team and the security team, since the members of the security team might not be specialists of the product.

In order to bridge this gap, we’ve have someone we call the ‘Security Champion’ in each product team. The Security Champion of the product team is responsible for maintaining the safety of the product and conducting vulnerability assessments.

All Security Champions (from different product teams) directly work with the Platform Security Team and share knowledge and experiences with each other. They also share the knowledge of the Platform Security Team back with the members of the product teams.

III: Patching up 

When a vulnerability is detected in a product, patches are created for all the versions that the issue exists in. If the severity of the vulnerability is catastrophic, these patches will be released to all customers immediately. If the severity is not catastrophic, we aggregate all patches developed during the month and release the lot at the end of the month as a security bulletin.

When a patch is ready, it’s sent out through WSO2 Update Manager (WUM), added to wso2.com/security-patch-releases and publicly announced. Every version of any given product supported by WUM will receive the patches automatically. Note that unless the product is supported by WUM, security patches are publicly released only for the very latest version of the products.

Moving forward, we’ve started recording this in Documentation at docs.wso2.com/display/Security/Security+Advisories for the sake of preserving more patch information. This effort is still recent but will add up over time.

IV: Responding to Vulnerability Reports

Technology gets updated every day and there are always new vulnerabilities and exploits discovered. We welcome contributions from our user community, developers, and security researchers to reinforce our product security. Over the years, a great many people – both customers and from the community -have helped us make our products the best they can be.

When someone reports a vulnerability, we try to verify the issue and respond to the reporter. If the vulnerability is a true positive, the patching process begins.

Generally, we do ask that the reporter refrains from publicly disclosing the vulnerability until we’ve patched it – this is to prevent anyone who might be vulnerable from being targeted.

We’re always looking for ways to make this easier. For example, we’ve set up wso2.com/security to serve as an easy, central point for our community to report issues. As time goes on,


 

References

[1] OWASP Secure Coding Practices https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

[2] Oracle Secure Coding Guidelines for Java http://www.oracle.com/technetwork/java/seccodeguide-139067.html

[3] SANS Secure Coding Guidelines https://www.sans.org/course/secure-coding-java-jee-developing-defensible-applications

[4] Static Code Analysis for Java using FindBugs Plugin and Identifying Security Bugs with FindSecurityBugs Plugin
http://tharindue.blogspot.com/2016/06/static-code-analysis-for-java-using.html

[5] OWASP Dependency Check CLI – Analyzing Vulnerabilities in 3rd Party Libraries http://tharindue.blogspot.com/2016/10/owasp-dependency-check-cli-analyzing.html

[6] Checking vulnerabilities in 3rd party dependencies using OWASP Dependency-Check Plugin in Jenkins https://medium.com/@PrakhashS/checking-vulnerabilities-in-3rd-party-dependencies-using-owasp-dependency-check-plugin-in-jenkins-bedfe8de6ba8#.ipu0b8u4o

[7] Dynamic Scanning with OWASP ZAP for Identifying Security Threats https://medium.com/@PrakhashS/dynamic-scanning-with-owasp-zap-for-identifying-security-threats-complete-guide-52b3643eee04#.nyy1fwiok

[8] Automating the boring stuff in development using ZAP and Jenkins : Continuous Integration
https://medium.com/@PrakhashS/automating-the-boring-stuffs-using-zap-and-jenkins-continues-integration-d4461a6ace1a#.jtknrzajt

Keep your WSO2 products up to date with WUM

We at WSO2 continuously improve our products with bug fixes, security fixes , and various other improvements. Every major release of our WSO2 products is followed by a series of periodic dot releases that roll up all these recent updates.

We want everyone evaluating, developing on, or preparing WSO2 products for production deployments to have the best experience and especially not trip over a bug that has already been fixed! WSO2 Update Manager (WUM) makes this process much easier – now you can get access to latest updates as and when we release them.

What is WUM?

WUM is a command-line application which allows you to keep WSO2 products up-to-date. It determines which updates are new and relevant for your WSO2 products, downloads and installs them. WUM builds you a new product distribution with all the relevant updates; this distribution is ready to deploy into your development or production environment. Note that WUM will not push updates into production: you have full control over when and how you want to such pushes to occur.

Updates are available for the following products and versions. We are in the process of adding more products and versions.

  • WSO2 Enterprise Service Bus 5.0.0
  • WSO2 Enterprise Service Bus 4.9.0
  • WSO2 Enterprise Service Bus Analytics 5.0.0
  • WSO2 API Manager 2.0.0
  • WSO2 API Manager Analytics 2.0.0

Note that fresh WSO2 product distributions downloaded via the wso2.com site or using WUM, are licensed under Apache 2. However, product distributions built by WUM with updates installed are licensed under WSO2 Update License 1.0. The updated product distributions are not licensed for use in production without a valid WSO2 subscription. Visit http://wso2.com/support for more information about WSO2 subscription.

Installing WUM

You can download WUM from http://wso2.com/wum. Download the binary release suitable for your operation system and processor architecture. Binary distributions are available for Linux, Mac OS, and Windows (follow installation instructions available on the same download page).

Once the installation is completed, open a new terminal and execute the following command. You should get the help page with all the available commands.

$ wum

Initialize WUM with your WSO2 credentials

You need to have a WSO2 account to get started with WUM. If you don’t have an account with WSO2, please create one here.

$ wum init
You need a WSO2 account to start using wum.
Don't have one yet? Sign up at https://wso2.com/user/register

Please enter your WSO2 credentials to continue
Username: sameera@wso2.com
Password for 'sameera@wso2.com': 
Authenticating...
Done!

-- Welcome to WUM 1.0-beta --

* Find more information at http://wso2.com/wum
* Documentation: https://docs.wso2.com/display/ADMIN44x/Updating+WSO2+Products
* JIRA: https://wso2.org/jira/browse/WUM

What's next? Have a look at the following list of wum commands:

Add WSO2 products to your product repository
  wum search  Search WSO2 products         
  wum add     Add or download WSO2 products

Update WSO2 products available in your product repository
  wum check-update Check for new updates    
  wum update       Update your WSO2 products

Manage WSO2 products available in your product repository
  wum list      List WSO2 products           
  wum describe  Show details of WSO2 products
  wum delete    Delete WSO2 products

Use "wum [command] --help" for more information about a command.

Use "wum [command] --help" for more information about a command.

Local product repository

WUM maintains a local product repository in your machine where it stores WS02 product distributions. Whenever you update a product, WUM creates a new distribution with all these updates and stores it in this repository. By default, WUM creates this repository in the user’s home directory, but you have the option to change this as well.


Search and add WSO2 products

Before you update any WSO2 product, you need to add the product to your local product repository. You can either download a product or add an already downloaded product. If you are not sure about what product name to use, you can use the ‘wum search’ command. If you run the search command without any arguments, it will list all the WUM supported WSO2 products.

$ wum search
Connecting to WSO2 Update...

wso2esb-5.0.0 - Enterprise Service Bus
wso2esb-analytics-5.0.0 - Enterprise Service Bus Analytics
wso2am-2.0.0 - API Manager
wso2am-analytics-2.0.0 - API Manager Analytics
wso2esb-4.9.0 - Enterprise Service Bus

What's next?
  use "wum add <product>" to download a product
  e.g "wum add wso2esb-5.0.0" to download Enterprise Service Bus 5.0.0

You can also search using keywords.

$ wum search api manager
Connecting to WSO2 Update...

wso2am-2.0.0 - API Manager
wso2am-analytics-2.0.0 - API Manager Analytics

What's next?
  use "wum add <product>" to download a product
  e.g "wum add wso2am-2.0.0" to download API Manager 2.0.0

If you look at the examples given in the “what’s next” section, you would see the expected format of the product name to use. Use the following command if you want to add the latest version of wso2esb.

$ wum add wso2esb
Connecting to WSO2 Update...
The following product(s) will be downloaded.
  wso2esb-5.0.0.zip
After this operation, 218.1MB of additional disk space will be used.
Do you want to continue? [Y/n]
Downloading product wso2esb-5.0.0... [2.4MB/218.1MB]

If you want to add a specific version of wso2esb then you need to specify the version as follows:

$ wum add wso2esb-4.9.0

You also have the option to add an already downloaded product distribution using the add command.

$ wum add --file ~/user/wso2/dist/wso2esb-5.0.0.zip

Now try the following command to see the list of add products.

$ wum list

Update WSO2 products

As I’ve explained earlier, WUM creates a new product zip file with all the updates installed. But if you want to check whether updates are available before you attempt to update, you can use the “check-update” command as follows.

$ wum check-update wso2esb-4.9.0
Connecting to WSO2 Update...

Checking for latest updates for wso2esb-4.9.0...
96 updates are available
[WARNING] 11 critical security updates. WSO2 strongly recommends that you install these updates now.

What's next?
  use "wum update" to install latest updates

Now let’s update the products using the “update” command

$ wum update wso2esb-4.9.0
Connecting to WSO2 Update...

Validating your subscription status for product wso2esb...
[WARNING] Your credentials are not associated with an active WSO2 subscription.
Please remember that updates are not licensed for use in production without a valid WSO2 subscription.
See: http://wso2.com/wum for more details.

Updating wso2esb-4.9.0...
96 updates are available
Downloading updates...
Installing updates...
Preparing update summary...
Building updated product...
Update summary:
  Installed updates: 96
  * [WARNING] WSO2 strongly recommends to use this updated product for production as soon as possible.
  Security updates: 11
  Updated Product: /Users/sameera/.wum-wso2/products/wso2esb/4.9.0/wso2esb-4.9.0.1472889183206.zip
  * More information about updates are available inside the above product archive
  Update summary(pdf): (product-archive)/updates/summary-2016-09-08T19-23-50/update-summary-1472889183206.pdf
  * Configuration changes are available
  Change log: (product-archive)/updates/summary-2016-09-08T19-23-50/change-log-1472889183206.txt

What's next?
  use "wum list [<product-pattern>]" to list products in your local repository
  use "wum describe [<product-pattern>]" to get more details of products

If you don’t have an active WSO2 subscription for wso2esb, you would get a warning as above. This means you can use the updated product for your development activities, but you are not supposed to use it in production.

Once the update is completed, you should be able to see the updated product files in your local repository. We version updated product distributions using the timestamp of the latest update.

$ wum list
Product        Updated              Filename                       
wso2am-2.0.0   08 Sep 16 18:01 IST  wso2am-2.0.0.1473322072976.zip 
wso2am-2.0.0   -                    wso2am-2.0.0.zip               
wso2esb-4.9.0  08 Sep 16 19:24 IST  wso2esb-4.9.0.1472889183206.zip
wso2esb-4.9.0  -                    wso2esb-4.9.0.zip

Manage your product repository

There are several WUM commands available for you to manage your local product repository. The “wum list” command lists all the available products. If you want to get more information about a product, you can use the “wum describe” command. It gives you the location of the product file and update history, among other information.

$ wum describe wso2esb-4.9.0.1472889183206.zip
Filename:  wso2esb-4.9.0.1472889183206.zip
Product Name:  wso2esb
Product Version: 4.9.0
Kernel Version:  4.4.1
Last Updated Time: 2016-09-08T19:23:55+05:30
Product File Path: /Users/sameera/.wum-wso2/products/wso2esb/4.9.0/wso2esb-4.9.0.1472889183206.zip
No Of Times Updated: 1
Installed Updates: 96
Update History:
 Date                       Installed Updates  Username        
 2016-09-08T19:23:55+05:30  96                 sameera@wso2.com

In case you want clean up your product repository by deleting files, you can use the “wum delete” command.

$ wum delete wso2esb-4.9.0
The following product file(s) will be deleted.
  wso2esb-4.9.0.1472889183206.zip
  wso2esb-4.9.0.zip
Do you want to continue? [y/N]

The above command will delete all the wso2esb-4.9.0 product files including the updated files. List, describe and delete commands accepts a product pattern; therefore you can filter out products by given a particular pattern.


We’ve released the Beta version of WUM. Try it out and let us know what you think.