[Article] WSO2 Carbon architecture: A high-level understanding and how it works

Applies to

Table of contents

• Introduction
• The old architecture (pre 4.0.0)
• The new architecture
• How server starts
• How tomcat starts
• How requests get dispatched
• Conclusion

Introduction

WSO2 Carbon is the base platform for all products of WSO2. By leveraging the OSGi [1] technology, the WSO2 Carbon architecture is designed in such a way that it's highly extensible, dynamic, and flexible. Over the years, the existing carbon platform was used to build a different set of products at WSO2 and helped to implement many solutions. The platform has gained maturity over time. There are new improvements, and features are added with each major release. The new family of products that are released now are based on Carbon version 4.0.0 and above.

The old architecture (pre 4.0.0)

In earlier releases, for example in version 3.2.0, carbon was deployed as a webapp in the embedded tomcat instance. The requests for carbon were handled using a concept called “Bridge Servlet” [2]. More details about “Bridge Servlet” can be found in the Equinox starter guide. The overall architecture and the request dispatching model of carbon server (pre 4.0.0) is described in the following image.

Figure 1

As shown in the above image, carbon was running entirely as a webapp within tomcat. Bridge Servlet was the one that initialized the OSGi framework. Given the nature of this, carbon could be deployed in any other servlet container or application server available. The entry point for all requests coming into carbon is Bridge Servlet.

When carbon is started, all incoming requests are directed into carbon webapp, which is then handled by the Bridge Servlet. It act as a connector/bridge for the external environment and the Carbon OSGi environment. All these requests coming into Brigde Servlet are then delegated into the OSGi HttpService [3]. All the required contexts, like /services, /carbon, etc. are registered in the OSGi HttpService, and Bridge Servlet will forward incoming requests to those appropriate registered context. The requests for other webapp contexts are dispatched by tomcat in the usual way.

However, as illustrated in the above image, that tomcat is not in the OSGI environment. Moreover, since carbon was deployed as a webapp, the OSGI environment (related libraries) were visible only to carbon webapp. Other webapps deployed alongside carbon (e.g. WSO2 Application Server [4]) cannot see these libraries, and APIs (such as CarbonContext API [5]), provided by the carbon OSGI environment. As a result, the libraries (jars) had to be duplicated for other webapps for their usage. This was a fundamental issue observed with earlier carbon releases.

The new architecture

There is a significant architectural change introduced in the Carbon 4.0.0 based release. The main change was bringing tomcat into the OSGi environment. After this change, no bridging mechanism is used to connect external requests to the carbon OSGi framework. The embedded tomcat library was created as an OSGI bundle and brought into the carbon OSGi environment. The carbon application now runs purely within the OSGI environment. The following image explains the new architecture in detail.

Figure 2

As shown in the above image, tomcat now sits entirely within the OSGi environment in contrast to what we saw in the earlier image. The embedded tomcat is now another OSGi bundle like other bundles.

When the carbon server starts and when tomcat run-time is initialized, a root context (/) webapp is deployed. This webapp registers a servlet called "Delegation Servlet", which will basically listen to all requests (/*) coming to the root context.

Once the server is properly started and transports are initialized, the requests will then be dispatched to this delegation servlet, which will then pass them to OSGi HTTPService [3], which in-turn will route them to the correct context. The entry point for requests into the OSGi environment is by using this OSGi HTTPService. How the requests gets dispatched is explained later in this article. As you can, since tomcat now resides within the OSGi environment, other webapps can also now see the carbon OSGi related libraries in their class path. This is because tomcat is just another bundle and the parent class-loader of tomcat is now the BundleClassLoader. So whenever a webapp (WebappClassLoader) needs a lib/classes from other OSGi bundles, it can be delegated by its parent (TomcatClassLoader), which in-turn can get it from its parent, which is the tomcat's BundleClassLoader. This eliminates the need for jar duplication in other webapps.

Even though carbon now is not deployed as a webapp, there will be a requirement to deploy carbon on other application servers as webapp. But, at present, this is strongly discouraged by WSO2. A post explaining about how to achieve the above requirement can be found here [6].

How server starts

In an OSGI environment, we cannot guarantee the starting orders of bundles. But in carbon, we need to have some bundle starting order. One of the main reason for this is the dependency on Apache Axis2 [7]. The Axis2 runtime should start before the axis2modules and axis2services can be deployed and the http transports can only be started when axis2 runtime is initialized properly. The following image shows a high level view of the starting sequence of major components in a plain WSO2 carbon standalone server.

Figure 3

Let us now look into each of the above steps in detail.

1. Using the server startup script (wso2server.sh | .bat) the server startup command can be issued, which will call the bootstrap module to load. This will basically call the Main class of the server. (org.wso2.carbon.server.Main).
2. The server module will do the actual server launching. Before invoking the OSGi framework launch, it will first invoke the server extensions tasks.
3. The server extensions mainly contains tasks such as
   1. DefaultBundleCreation: This will create OSGi bundles out of regular jar files found in ${carbon.home}/repository/components/lib directory.
   2. SystemBundleExtensionCreation: This will read the extension folder ${carbon.home}/extensions and create system extension bundles.
   3. Log4jPropertyFileFragmentBundleCreation: This will create a fragment bundle for log4j.properties file and place it in the ${carbon.home}/repository/components/dropins directory to be used in OSGi enironment.
   4. DropinsBundlesDeployment: This will install bundles found in dropins directory to the OSGi environment by adding them to the bundles.info file.
   5. PatchInstallation: This will basically install the pathced jar files by copying them to ${carbon.home}/repository/components/plugins directory.
4. Once the above tasks are completed, the server module will launch the OSGi framework. This will load the OSGi System bundle, which will then load the OSGi Simple Configurator Bundle. This bundle is responsible for reading the bundle.info file and will populate the bundle loading list and load bundles according to the start order.
5. During this process, many bundles activators will be called. Some of the important bundles of interest are the User Core and Registry Core bundle. They will basically initialize the user management and registry aspect of the server. After initializing, the respective OSGi services will be published, so the interested party can find and use them.
6. Only after the above step, the initialization of carbon kernel will start. This will first initialize Axis2 run-time. After initializing axis2, then the artifact deployment process will take place. This will deploy axis2modules and then axis2services. Since now the Axis2 run-time is initialized, this will also deploy any webapps available in the server.
7. After the artifacts have been deployed, the carbon kernel will now initialize and start transports and make the system ready to accept requests.
8. Once the transports are up, the final part of the server start-up is to start the UI framework and load all the UI-related resources and make the carbon management console accessible.

How tomcat starts

How the embedded tomcat in carbon starts is not explicitly mentioned in the above section, it is important to understand this as well.

When the carbon OSGi framework is initialized, and during all the bundle activation process, the bundle responsible for starting tomcat (org.wso2.carbon.tomcat) will also get called. This will call the OSGi BundleActivator class of that bundle (which is org.wso2.carbon.tomcat.internal.TomcatBundleActivator [8]). This activator will first create an instance of ServerManger. The ServerManger is a manager class for configuring, initializing, starting, and stopping a tomcat instance within carbon.

During ServerManger's initialization, it will first read and parse the tomcat server configuration file ${carbon.home}/repository/conf/tomcat/catalina-server.xml, which will be used for configuring the embedded tomcat instance. Then it will start the tomcat instance. A point to note here is that starting of tomcat here only enables activities such as add new webapps, etc. However, this will not start the tomcat servlet transport and open up http/s ports. This has been purposefully introduced to delay the starting of the http transports ports. This is because the server needs to come to a stable stage, such as axis2 run-time is initialized as ready, etc. before it can start serving requests. The starting of the transport happens during the server start-up finalization phase, which makes sure that all required services are started and the server is now at a stable stage to accept requests.

How requests get dispatched

The requests for carbon and its related services are handled using a delegated servlet registered with root context webapp. The root context webapp is the first webapp to be deployed when tomcat run-time starts. The repository location for this webapp is found in “${carbon.home}/repository/conf/tomcat/carbon”. Tomcat run-time can now see this servlet like any other servlets or webapps deployed in it. This servlet is registered to listen to the request with context patterns “/*” and *.jsp”. The servlet registration details are defined in the web.xml of the root webapp.

During run-time, the requests are then passed to OSGI HttpService, which then route them to the correct context. All the required contexts should be registered in the OSGi HttpService for this to work properly. The following image shows the run-time representation for requests dispatching.

Figure 4

Following are the major contexts that get registered with HTTPService during server start-up. Each of the context is well defined to serve a particular purpose.

1. /carbon – Requests for Carbon UI framework (org.wso2.carbon.ui), which includes the management console requests as well. This context with all the patterns (*.jsp) is registered with the required servlets by the CarbonUIServiceComponent during its activation process.
2. /services – Requests for Axis2 services that are handled by CarbonServlet (org.wso2.carbon.core). These requests are then handed over to Axis2, which will handle them according to the artifacts deployed. This context is registered for CarbonServlet, which is responsible for all the HTTP and HTTPS requests for axis2 runtime within carbon. This servlet registration is done by the CarbonServerManger during its starting process by the carbon kernel.
3. /fileupload – Requests for file upload servlet. This context is registered for FileUploadServlet, which handles the upload requests for carbon, and this registration is carried out by the CarbonUIServiceComponent during its activation process.
4. /filedownload – Requests for file download servlet. In here as well CarbonUIServiceComponent does the registering of this context for FileDownloadServlet, which handles the http download requests in carbon.

Conclusion

The new architecture of WSO2 Carbon platform based on version 4.0.0 and over has many advantages; it differs from the previous architecture by way of requests getting dispatched in carbon based on the new architecture. Moreover, given that it leverages the OSGi technology and follows a modular architecture as its base, enterprises can benefit in numerous ways as this makes the platform extensible, dynamic, and flexible.

References

[1] https://eclipse.org/equinox/server/http_in_container.php
[2] https://www.osgi.org/javadoc/r4v42/org/osgi/service/http/HttpService.html
[3] https://wso2.com/products/application-server/
[4] https://docs.wso2.org/display/Carbon420/CarbonContext+API
[5] https://pradeepfernando.blogspot.com/2012/09/running-wso2-carbon-as-web-app-apache.html
[6] https://axis.apache.org/axis2/java/core/
[7] https://svn.wso2.org/repos/wso2/carbon/kernel/branches/4.2.0/core/org.wso2.carbon.tomcat/4.2.0/src/main/java/org/wso2/carbon/tomcat/internal/TomcatBundleActivator.java