Tag Archives: software

Implementing an Effective Deployment Process for WSO2 Middleware


Image reference: https://www.pexels.com/photo/aerospace-engineering-exploration-launch-34521/

At WSO2, we provide middleware solutions for Integration, API Management, Identity Management, IoT and Analytics. Running our products on a local machine is quite straightforward: one just needs to install Java, download the required WSO2 distribution, extract the zip file and run the executable.

This provides a middleware testbed for the user in no time. If the solution needs multiple WSO2 products, those can be run on the same machine by changing the port-offsets and configuring the integrations accordingly.

This works very well for trying out product features and implementing quick PoCs. However, once the preliminary implementation of the project is done, a proper deployment process is needed for moving the system to production. 

Any software project needs at least three environments for managing development, testing, and the live deployments. More importantly, a software governance model would be needed for delivering new features, improvement, bug fixes and managing the overall development process.

This becomes crucial when the project implements the system on top of a middleware solution. Both middleware and application changes will need to be delivered. There might be considerable amounts of prerequisites, artifacts and configurations. Without having a well-defined process, it would be difficult to manage such projects efficiently.

A High-Level Examination

One would have to consider the following points would need to be considered when implementing an effective deployment process:

  • Infrastructure

WSO2 middleware can be deployed on physical machines, virtual machines and on containers. Up to now most deployments have been done on virtual machines.

Around 2015, WSO2 users started moving towards container-based deployments using Docker, Kubernetes and Mesos DC/OS. As containers do not need a dedicated operating system instance, this cuts down resource requirements for running an application – in contrast to a VM. In addition, the container ecosystem makes the deployment process much easier using lightweight container images and container image registries.

We provide Puppet Modules, Dockerfiles, Docker Compose, Kubernetes and Mesos (DC/OS) artifacts for automating such deployments.

  • Configuration Management

The configuration for any WSO2 product can be found inside the relevant repository/conf folder. This folder contains a collection of configuration files corresponding to the features that the product provides.

The simplest solution is to maintain these files in a version control system (VCS) such as Git. If the deployment has multiple environments and a collection of products, it might be better to consider using a configuration management system such as Ansible, Puppet, Chef or Salt Stack for reducing configuration value duplication.

We ship Puppet modules for all WSO2 products for this purpose.

  • Extension Management

WSO2 middleware provides extension points in all WSO2 products for plugging in required features.

For example, in WSO2 Identity Server a custom user store manager can be implemented for connecting to external user stores. In the WSO2 Integration products, handlers or class mediators can be implemented for executing custom mediation logic.  Almost all of these extensions are written in Java and deployed as JAR files. These files will simply need to be copied to the repository/components/lib folder or the repository/components/dropins folder if they are OSGi compliant.

  • Deployable Artifact Management

Artifacts that can be deployed in repository/deployment/server folder fall under this category. For, example, in the ESB, proxy services, REST APIs, inbound endpoints, sequences, security policies can be deployed in runtime via the above folder.

We recommend that you create these artifacts in WSO2 Developer Studio (DevStudio) and package them into Carbon Archive (CAR) files for deploying them as collections. WSO2 DevStudio provides a collection of project templates for managing deployable files of all WSO2 products. These files can be effectively maintained using a VCS.

These files can be effectively maintained using a Version Control System.

  • Applying Patches/Updates

Patches were applied to a WSO2 product by copying the patch<number> folder which is found inside the patch zip file to the repository/deployment/patches/ folder.

We recently introduced a new way of applying patches for WSO2 products with WSO2 Update Manager (WUM). The main difference of updates, in contrast to the previous patch model, is that fixes/improvements cannot be applied selectively; it applies all the fixes issued up to a given point using a CLI. This is the main intention of this approach.

  • Lifecycle Management

In any software project it is important to have at least three environments – one for managing development, one for testing and one for production deployments.  New features, bug fixes or improvements need to be first done in the development environment and then moved to the testing environment for verification. Once the functionality and performance are verified the changes can be applied in production (as explained in the “Rolling Out Changes”) section.

The performance verification step might need to have resources identical to the production environment for executing load tests. This is vital for deployments where performance is critical.

With our products, changes can be moved from one environment to the other as a delivery.  Deliveries can be numbered and managed via tags in Git.

The key advantage of using this approach is the ability to track, apply and roll back updates at any given time.

  • Rolling Out Changes

Changes to the existing solution can be rolled out in two main methods:

1. Incremental Deployment (also known as Canary Release).

The idea of this approach is to incrementally apply changes to the existing solution without having to completely switch the entire deployment to the new solution version. This gives the ability to verify the delivery in the production environment using a small portion of the users before propagating it to everyone.

2. Blue-Green Deployment

In Blue-Green deployment method, the deployment is switched to the newer version of the solution at once. It would need an identical set of resources for running the newer version of the solution in parallel to the existing deployment until the newer version is verified. In case of failure, the system can be switched back to the previous version via the router. Taking such approach might need a far more thorough testing procedure compared to the first approach.

Deployment Process Approach 1

This illustrates the simplest form of executing a WSO2 deployment effectively.

In this model the configuration files, deployable artifacts and extension source code are maintained in a version control system. WSO2 product distributions are maintained separately in a file server. Patches/updates are directly applied to the product distributions and new distributions are created. The separation of distributions and artifacts allows product distributions to be updated without losing any project content.

As shown by the green box in the middle, a deployable product distribution is created, combining the latest product distributions, configuration files, deployable artifacts and extensions. Deployable distributions can be extracted on physical, virtual machines or containers and run. Depending on the selected deployment pattern, multiple deployable distributions will need to be created for a product.

In a containerized deployment, each deployable product distribution will have a container image. Depending on the containerized platform, a set of orchestration and load balancing artifacts might also be used.

Deployment Process Approach 2

In the second approach, a configuration management system has been used for reducing the duplication of the configuration data and automating the installation process.

Similar to the first approach, deployable artifacts, configuration data and extension source code are managed in a version control system. Configuration data needs to be stored in a format that is supported by the configuration management system.

For an example, in a Puppet configuration, data is either stored in manifest files or Hiera YAML files. Deployable WSO2 product distributions are not created. Rather, that process is executed by the configuration management system inside a physical machine, virtual machine or in a container at the container build time.

In conclusion

Any of the deployment approaches we’ve spoken about above can be followed with any infrastructure. If a configuration management system is used, it can be used for installing and configuring the solution on virtual machines and as well as on containers. The main difference with containers is that configuration management agent will only be triggered at the container image build time. It may not be run in the when the container is running.

If a configuration management system is used, it can be used for installing and configuring the solution on virtual machines and as well as on containers. The main difference with containers is that configuration management agent will only be triggered at the container image build time. It may not be run in the when the container is running.

At the end of the day, a proper deployment process is essential. For more information and support, please reach out to us. We’d be happy to help.

How we handle security at WSO2

A Proactive Strategy for Security Management

Any decent software development organization generally has a well-defined set of policies and procedures for security management.

At WSO2, we – as in, the Platform Security Team – constantly collaborate with other product teams, customers and external security researchers to manage overall security of all WSO2 product. In this post, we’d like to talk about how we do this.


Part One: in the realm of code

code-944504_1280

I: Designing for security

The first stage of software design is the gathering of requirements. In open source software, we tend to use third-party code quite a bit – it’s how open source works: we stand on the shoulders of giants.
However, we can’t simply use what code we think is suitable.

The first check comes here. At WSO2, if we identify any kind of third-party code to be used, we need it to be first approved by the Engineering Management group, who are an internal group of seasoned architects who function at a directorial level. For us, security comes as a first priority, not as an afterthought.

The next set of checks come in the design phase. What are the communication protocols being used? How secure are they? Where is the data stored, and how? What endpoints are we exposing to the public? We go through a series of use cases to identify where this design can be broken, and work with the product design team to integrate our security concerns from the start.

II: Review, rinse, repeat

The next part is obvious: every developer is responsible for writing clean code [1, 2, 3].

Code written by each developer goes through a process of code quality reviewing overseen by members of the relevant product team and the Platform Security Team. When submitting the code for reviewing, the developer has to submit the static code analysis reports – generated using tools like FindSecBugs [4]. This is a mandatory security check in the reviewing process. Only upon fixing all issues spotted in the first pass is code is merged to the repository.

III: Testing with the automated grindhouse

At WSO2, we use Jenkins quite a lot for automating the build process. It builds individual components; it packages components together; it constantly builds and re-builds.

A large part of our security testing is integrated right into this process. Jenkins first performs the OWASP Dependency Check [5, 6], which analyzes the project dependencies and produces vulnerability reports. Even after the selection process in the first stage is complete, there can be some vulnerabilities that we haven’t spotted – especially if they’ve only been discovered extremely recently.

Next, Jenkins uses FindSecBugs as a plugin; during each automated build cycle, it checks individual components and generates vulnerability reports, which are in turn submitted to the security team for review.

Jenkins also uses the OWASP Zed Attack Proxy for dynamic code analysis [7, 8]. During the dynamic security analysis, the entire URL tree of the product is scanned and well-known attacks and exploits are automatically performed; the results are reported. These reports, too, are investigated by the respective product team as well as the Platform Security Team.

Once the testing is complete and a product is ready to be released, the respective product team has to receive security clearance from the Platform Security Team. If any known vulnerabilities are still listed in the reports, the product team has to justify to us the existence of the reported vulnerability – a pretty hard job.

We find that developers may write code following all the best security practices, but when the code is merged together, it might still open up a vulnerability because of how everything integrates together.


 Part Two: when humans happen

astronaut-and-robonaut-shake-hands

I: Preparing for the real world

There’s a saying: no battle plan survives contact with the customer. Although security standards and processes are followed to the letter, our products have to run in the real world.

One of the most important things is building awareness. We put together a set of deployment patterns, security recommendations, and best practices to be followed when deploying our products; we also conduct public webinars for making awareness in security related topics for WSO2 users, which are available at wso2.com/library/webinars.

II: Building internal Champions

Sometimes there is a gap between the product team and the security team, since the members of the security team might not be specialists of the product.

In order to bridge this gap, we’ve have someone we call the ‘Security Champion’ in each product team. The Security Champion of the product team is responsible for maintaining the safety of the product and conducting vulnerability assessments.

All Security Champions (from different product teams) directly work with the Platform Security Team and share knowledge and experiences with each other. They also share the knowledge of the Platform Security Team back with the members of the product teams.

III: Patching up 

When a vulnerability is detected in a product, patches are created for all the versions that the issue exists in. If the severity of the vulnerability is catastrophic, these patches will be released to all customers immediately. If the severity is not catastrophic, we aggregate all patches developed during the month and release the lot at the end of the month as a security bulletin.

When a patch is ready, it’s sent out through WSO2 Update Manager (WUM), added to wso2.com/security-patch-releases and publicly announced. Every version of any given product supported by WUM will receive the patches automatically. Note that unless the product is supported by WUM, security patches are publicly released only for the very latest version of the products.

Moving forward, we’ve started recording this in Documentation at docs.wso2.com/display/Security/Security+Advisories for the sake of preserving more patch information. This effort is still recent but will add up over time.

IV: Responding to Vulnerability Reports

Technology gets updated every day and there are always new vulnerabilities and exploits discovered. We welcome contributions from our user community, developers, and security researchers to reinforce our product security. Over the years, a great many people – both customers and from the community -have helped us make our products the best they can be.

When someone reports a vulnerability, we try to verify the issue and respond to the reporter. If the vulnerability is a true positive, the patching process begins.

Generally, we do ask that the reporter refrains from publicly disclosing the vulnerability until we’ve patched it – this is to prevent anyone who might be vulnerable from being targeted.

We’re always looking for ways to make this easier. For example, we’ve set up wso2.com/security to serve as an easy, central point for our community to report issues. As time goes on,


 

References

[1] OWASP Secure Coding Practices https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

[2] Oracle Secure Coding Guidelines for Java http://www.oracle.com/technetwork/java/seccodeguide-139067.html

[3] SANS Secure Coding Guidelines https://www.sans.org/course/secure-coding-java-jee-developing-defensible-applications

[4] Static Code Analysis for Java using FindBugs Plugin and Identifying Security Bugs with FindSecurityBugs Plugin
http://tharindue.blogspot.com/2016/06/static-code-analysis-for-java-using.html

[5] OWASP Dependency Check CLI – Analyzing Vulnerabilities in 3rd Party Libraries http://tharindue.blogspot.com/2016/10/owasp-dependency-check-cli-analyzing.html

[6] Checking vulnerabilities in 3rd party dependencies using OWASP Dependency-Check Plugin in Jenkins https://medium.com/@PrakhashS/checking-vulnerabilities-in-3rd-party-dependencies-using-owasp-dependency-check-plugin-in-jenkins-bedfe8de6ba8#.ipu0b8u4o

[7] Dynamic Scanning with OWASP ZAP for Identifying Security Threats https://medium.com/@PrakhashS/dynamic-scanning-with-owasp-zap-for-identifying-security-threats-complete-guide-52b3643eee04#.nyy1fwiok

[8] Automating the boring stuff in development using ZAP and Jenkins : Continuous Integration
https://medium.com/@PrakhashS/automating-the-boring-stuffs-using-zap-and-jenkins-continues-integration-d4461a6ace1a#.jtknrzajt

Connected Health – Reinventing Healthcare with Technology

Demands for more personalized and convenient services from healthcare providers has steadily increased during the past decade. Increase in populations, life expectancy, and the advancement of technology are a few key contributors to this uptick in demand. These demands have resulted in creating a global eHealth market that is supposed to reach $308 billion by 2022, as predicted by Grand View Research INC.

The essence of a connected healthcare business is to deliver an efficient, effective service to its users by connecting disparate systems, devices, and stakeholders. It aims to automate most tasks and eliminate human error, trigger intelligent events for the hospitals and other stakeholders, and provide medical information via a range of devices at various locations. By becoming a connected ecosystem, hospitals have the opportunity to reduce costs, increase revenue, as well as offer a high-quality service to patients.

The success of a connected healthcare business though depends on how the enterprise will look to address key challenges via comprehensive solutions.

In the white paper “Connected Health Reference Architecture” Nuwan Bandara, a solutions architect at WSO2, discusses the significance of creating a connected healthcare system and explains how a middleware platform can be used to address each and every challenge faced at implementation.

Screen Shot 2016-04-26 at 3

One of the key challenges he highlights is the ability to deliver aggregated information without  any latency issues between sources. In order to overcome this, you would need a centralized system that enables smooth integration of devices, services, and workflows. The use of multiple devices that take various measurements in different formats makes it a bit more difficult compared to other connected ecosystems; however, this can be addressed by consolidating the gathered data, and making it easily accessible to various services and applications from different locations.

Given that all this data is private information, it is vital to have fool proof security measures in place as well to restrict access only to authorized personnel, Nuwan notes.

Furthermore, it is important for hospitals to be geared to manage high capacities during crisis situations. If the system is unable to cope with high loads during these times, the system will crash and disrupt all workflows. Hospitals overcome this by equipping their systems with elastic scaling to handle high loads.

To learn more about the Connected Health reference architecture, download and read the white paper here .

WSO2Con Asia 2016 – highlights, pictures and turkey tweets!

Last week we concluded WSO2Con Asia 2016 with a bang! Over 250 attendees from across the region joined us, making it our biggest user conference to date.

We had inspiring keynotes, customer stories and technical sessions ranging from analytics and API management to cloud and integration. We also had lots of fun like twitter competitions and after parties.

Here’s a recap on some of our most memorable moments…

Day 1 was dedicated to tutorials. The sessions provided hands-on experience and deep dives into key WSO2 products. The ones on microservices and IoT were specially popular.

Tutorials

The official start of the conference however, was day 2 where the punchy sounds of Sachintha and the Beat Drummers kicked things off at the opening ceremony.

WSO2ConAsia-2016Opening

This was followed by the opening keynote by WSO2 Founder, CEO and Chief Architect Dr. Sanjiva Weerawarana. In his usual flair Sanjiva talked about WSO2’s progress in the past 10 years and how the WSO2 platform is being used by customers across industries such as  transport, government, entertainment, mobile and more. He also spoke about WSO2’s vision for the connected enterprise and its future in the middleware industry.

Sanjiva

Other keynotes for the day included an engaging presentation on Vega, the high performance electric sports car being developed in Sri Lanka. Dr. Harsha Subasinghe, the president and CEO of Codegen, and Dr. Beshan Kulapala, a research scientist at Codegen highlighted the challenges and opportunities in leading complex engineering projects to success.

Codegen

Dr. Frank Leymann, a director in the Institute of Architecture of Application Systems at University of Stuttgart, discussed loose coupling and its implications on microservice and cloud native architectures.

Frank Leymann

Isabelle Mauny, the vice president of product management at WSO2, gave the final keynote for the day on how analytics can improve customer experiences. Interestingly the phrase “Turkeys” was trending on our twitter dashboard during her talk!

The day continued with dedicated tracks running in parallel on cloud, API management, security and integration.

Several customers shared their experiences and how they have used the WSO2 Platform to effectively meet their technical and business goals.

Harshavardhan Mohanraj and Praveen Doddamani, technical leads at ZeOmega, spoke about how they leveraged the WSO2 platform to build a healthcare solution while improving component manageability and standardizing security in the API Management Track.

Zeomega

Ibrahim Khalil, a system integration analyst and team lead at Capgemini, spoke about how they leveraged their experience with United Nations agencies and built a vertical solution, enabled by WSO2 products, for UN organizations in the Cloud Track.

Ibrahim Khalili

Gina Keune, the team lead of integration and configuration at Royal Automobile Association, shared the challenges they faced and wins they celebrated when incrementally adopting SOA using WSO2.

Gina Keune

Charith De Silva, a lead architect at WSO2.Telco, introduced WSO2.Telco IDS which provides a fully Mobile Connect (OIDC) compliant solution for telcos embarking on a federated ID strategy.

Charith De Silva

This year we also hosted a special Strategy Forum for CxOs which saw attendees from companies such as Honeywell, NYU, John Keells Holdings, zMessenger, LOLC and more. The forum was led by WSO2 VP of Solutions Architecture Asanka Abeysinghe who spoke about the digital transformation of enterprise platforms.

Strategy Forum

The day ended with a networking event featuring smooth jazz tunes provided by Brown Sugar. Tasteful bites accompanied by cocktails coupled with lounge-like seating made it the perfect environment to catch up with industry experts and peers.

Networking event

The third and final day of the conference was also packed with insightful technical sessions. The tracks covered topics on governance, IT consumerization, analytics, devOps and app development.

The session on microservices attracted over a 100 of the attendees and proved to be one of the most popular talks of the day.

Azeez

Another popular talk was by Kiran Kumar, an enterprise architect at Wipro, who discussed a case study of a governance system where service governance meets API governance.

Kiran Kumar

The technical sessions came to an end with a closing keynote by Asanka and panel discussion on the benefits and effects of creating a digital enterprise.

Asanka Panel

In addition to the technical sessions, a team led by Srinath Perera, vice president of research at WSO2, showcased capabilities of the WSO2 Analytics platform by hooking up with Twitter to create a sentiment analyser. The project which tracked #wso2conasia was able to give valuable insights into sessions and popular topics. It also helped identify the most dedicated tweep, who walked away with a GoPro camera for his contributions.

Sentiment analysis

Sumedha Rubasinghe, director of API architecture at WSO2 also ran a project that combined  Google’s voice API with the soon to be released WSO2 IoT Server which displayed what speakers of each session were talking about.  This was just a sampling of what you can expect in the future with WSO2’s IoT Platform.

Voice Analyser

Source: Readme.LK

In true WSO2 style, WSO2Con Asia 2016 came to a close with a rocking after-party featuring the band Glory.

Party

In the coming weeks we’ll be sharing more details of the presentations, so stay tuned for that. In the meantime you can check out the slides for all sessions at the WSO2Con Asia website.