2012/10/22
22 Oct, 2012

Leveraging full features of Apache Tomcat in the WSO2 Application Server

  • Kishanthan Thangarajah
  • Senior Technical Lead - WSO2

Applies To

WSO2 Application Server 5.0.0

Table of Content

Introduction

Apache Tomcat is mainly used as a servlet container in WSO2 Carbon based servers. It runs in embedded mode in all servers. It also holds some useful J2EE features such as the ability to deploy webapps, JSP support, JNDI capabilities, JDBC data-sources, clustering, virtual hosting, and so on. The WSO2 Application Server uses the webapp hosting feature from Apache Tomcat to provide an easy way for users to host and manage their webapps. The following section explains how the other features of Apache Tomcat is made available in the WSO2 Application Server.

A standalone Apache Tomcat offers more useful features for webapp deployment and management such as clustering and session replication support for webapps. This feature is also available in the embedded mode. But when using an embedded mode of Tomcat in your servers, integrating this feature to work in your own environment is not a straight forward task, compared to a standalone instance. For example, enabling clustering and session replication is done through some changes to the server descriptor (server.xml) file. In the embedded mode a server descriptor file will not be available. To cater to these facilities, there are some custom changes made in the way Tomcat is embedded in WSO2 Carbon 4.0.0 based servers, making the WSO2 Application Server 100% Apache Tomcat compliant.

Current Limitations

The following are some limitations of using the embedded mode of Apache Tomcat, which were observed in WSO2 Carbon 3.x.x based servers.

  • It does not contain a server descriptor file (server.xml)
  • It does not support to include context descriptor(context.xml) for individual webapps as it is not getting loaded by the embedded server.
  • No support for global deployment descriptor file(web.xml)

Newly supported features

With the latest effort to move Apache Tomcat into WSO2 Carbon's OSGi environment and with changes in the way we embed Tomcat, we overcame the above mentioned limitations. Lets take a look at each of them in detail.

1. Support for server descriptor file

From WSO2 Application Server 5.0.0 onwards, embedded Apache Tomcat in the WSO2 Carbon kernel supports the server descriptor file (server.xml). This feature was not there in earlier releases as the same functionality was done programatically in the Carbon kernel. But this had some limitations on getting the full features supported by Tomcat such as creating new containers and new hosts for a server etc. So by adding this, we can get the full support similar to a standalone version.

The server descriptor file allows us to customize the Tomcat server configurations [1]. In this file, the top level component is Server. For a Tomcat instance, we can have at most one server component. The next level component is Service. There can be one or more Service components for a server component. We can define one or more containers, such as Context, Engine, Host and Cluster, in each of the Service components. For containers like Engine, Host and Cluster, there can be multiple sub components defined. A Tomcat valve is one such component. These valves come in handy when you want to work with incoming requests, before the request reaches the webapp. The valves can be inserted into the request processing pipeline of the Tomcat engine using the server descriptor file. This process can also be done programatically.

The main limitation of previous releases was, all of the above was done programatically. So if a new component or a container needed to be added, some code level changes were mandatory. But this was made easy, with the new changes to the way embedded Tomcat is integrated and the support for server descriptor file. The server descriptor file is located in {CARBON_HOME}/repository/conf/tomcat directory, with “catalina-server.xml” name. The following are the default settings in catalina-server.xml for tomcat in WSO2 Carbon 4.0.0 based servers.

<Server port="8005" shutdown="SHUTDOWN">
  <Service className="org.wso2.carbon.tomcat.ext.service.ExtendedStandardService" name="Catalina">
    
    <Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9763"
                bindOnInit="false"
                maxHttpHeaderSize="8192"
                acceptorThreadCount="2"
                maxThreads="250"
                minSpareThreads="50"
                disableUploadTimeout="false"
                connectionUploadTimeout="120000"
                maxKeepAliveRequests="200"
                acceptCount="200"
                server="WSO2 Carbon Server"
                compression="on"
                compressionMinSize="2048"
                noCompressionUserAgents="gozilla, traviata"
                compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg" 
                URIEncoding="UTF-8"/>
  
    
    <Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9443"
                bindOnInit="false"
                sslProtocol="TLS"
                maxHttpHeaderSize="8192"
                acceptorThreadCount="2"
                maxThreads="250"
                minSpareThreads="50"
                disableUploadTimeout="false"
                enableLookups="false"
                connectionUploadTimeout="120000"
                maxKeepAliveRequests="200"
                acceptCount="200"
                server="WSO2 Carbon Server"
                clientAuth="false"
                compression="on"
                scheme="https"
                secure="true"
                SSLEnabled="true"
                compressionMinSize="2048"
                noCompressionUserAgents="gozilla, traviata"
                compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
                keystoreFile="${carbon.home}/repository/resources/security/wso2carbon.jks"
                keystorePass="wso2carbon" 
                URIEncoding="UTF-8"/>

    <Engine name="Catalina" defaultHost="localhost"/>
      
      <Realm className="org.wso2.carbon.tomcat.ext.realms.CarbonTomcatRealm"/>
      <Host name="localhost" unpackWARs="true" deployOnStartup="false" autoDeploy="false" appBase="${carbon.home}/repository/deployment/server/webapps/">
          <Valve className="org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve"/>
          <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="combined" />
          <Valve className="org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve" threshold="600"/>
          <Valve className="org.wso2.carbon.tomcat.ext.valves.CompositeValve"/>
     </Host>
    </Engine>
  </Service>
</Server>

2. Support for context descriptor file

When it comes to webapps, its required to define Resources that are specific to themselves. This is done through context descriptor file(context.xml) [2] located in /META-INF directory in each webapp. If some resources are needed by all webapps that can be globally accessed, then these resources are defined in the server.xml as Global Resources. In previous releases of the WSO2 Application Server (i.e. 4.1.2), it didn't have the support for including webapp specific context descriptor files. This was due to a limitation in the embedded Tomcat API. However from 5.0.0 onwards, this limitation was overcome by modifying the embedded Tomcat API method for webapp handling in WSO2 Tomcat integration.

Further, with the support for context descriptor file in webapps, the WSO2 Application Server can now inherit the JNDI InitialContext implementation of Tomcat where we can define JNDI resources either globally of for individual webapps and use lookup and use them in our application implementations. You can refer to this blog post on how to register JNDI resources for webapps in WSO2 Application Server [3].

3. Support for global deployment descriptor file

Another limitation in embedded Tomcat is that, it does not support having a global deployment descriptor(web.xml) [4]. This too was due to the way embedded Tomcat handles adding webapps. So by overriding and changing API methods for handling root web.xml, the WSO2 Application Server now supports the global deployment descriptor file which acts as the default web.xml for all webapps. This will be the default web.xml for all webapps. So if a webapp doesn't include a deployment descriptor, this file will be used as the deployment descriptor for that webapp. In addition if we want to impose some default parameters for all webapps such as session timeout values, we can now do that with this root web.xml.

The above improvements make WSO2 Application Server a 100% Apache Tomcat compliant. Now whatever that can be done with a standard Tomcat can also be done with the WSO2 Application Server.

We will now move on to the next section which explains how the session replication feature of Apache Tomcat was integrated with the WSO2 Application Server.

Clustering and Http Session Replication

A standalone Tomcat server in a non-clustered environment does not offer support for high availability or fail-over. If the server failed unexpectedly, all the current active sessions get lost. And when the server is restarted, users will have to start all over again by entering all the lost data to resume their work. This is a limitation found in a standalone Tomcat server. Also, Handling high load requests or load balancing is another limitation observed in a non-clustered environment.

This is not an issue however, if the server is part of a cluster. In this case the server can provide both fail-over and scalability support. So when a node becomes unavailable, another sever node in the cluster can still serve the requests of users. Also when the load to one server node gets high, the load can be redirected to another node in the cluster by using load balancing techniques. This makes a cluster of Tomcat server nodes to be highly available and reliable. Also the cluster environment can scale based on the incoming load, by adding new nodes to the cluster. These features from a clustered environment provide users an uninterrupted service.

Http Session replication for webapps is a useful feature in a cluster environment. Apache Tomcat has inbuilt support for both clustering and session replication [5]. It uses Tribes [6] as the underlying communication framework to send session data and replication messages across the cluster. The session state in one node is replicated to all the other nodes in the cluster. So that when a different node becomes active, the relevant webapp is loaded with the current active session. The users are unaware of this, as they will continue to do their work without any interruption. The subsequent changes in the session will also be replicated to every other node.

The same feature is also available with an embedded mode of Tomcat. This can be leveraged and used in WSO2 Application Server as well. But as explained earlier, this is not a straight forward task.

Configurations (enabling and disabling cluster and session replication) can be changed via the catalina-server.xml configuration file. But there are some custom implementations on how to handle session replication in the WSO2 Application Server. The important change is the way the cluster communication framework is used - WSO2 Application Server already uses Apache Axis2 clustering [7] for Web services. In Apache Axis2 clustering, the undelying group communication framework is Tribes. Tomcat also uses this framework for communication. So when enabling clustering for both Apache Axis2 and Apache Tomcat, we will end up using two instances of Tribes framework running underneath. However there really is no point in having two instances for the same purpose. One instance of the Tribes frameworks can be used by both Apache Axis2 clustering and Apache Tomcat clustering to send cluster messages. A common cluster massage, which can satisfy both Apache Axis2 and Apache Tomcat clustering, should be used here. This change has been implemented and will be available from WSO2 Application Server 5.0.0 onwards.

In Apache Tomcat, it uses a valve(ReplicationValve) to initiate the session replication. The cluster sessions are managed by two managers in Tomcat. They are BackupManager and DeltaManager. The latter replicates session data to all nodes in the cluster where as the former replicates session data to only one backup node. The other nodes in the cluster knows the location of this backup node.

WSO2 has developed its own TomcatValve to start the session replication process and its own implementation of ClusterableSessionManager, which is responsible for handling sessions for a distributable webapp and sending session replication messages.

Following are the instructions to enable session replication in a WSO2 Application Server cluster. The Tomcat server descriptor file in WSO2 Carbon based servers is catalina-server.xml. This is equivalent to the server.xml file which is found in a standalone instance of Tomcat.

1. First enable clustering in Apache Axis2. We have to also enable clustering for the Catalina engine by adding the following in-between in catalina-server.xml.

<Cluster className="org.wso2.carbon.core.session.CarbonTomcatSimpleTcpCluster"/>

2. Add the following TomcatValve to which is responsible for starting the replication of session. This should be the first of the Tomcat valves.

<Valve className="org.wso2.carbon.tomcat.ext.valves.CarbonTomcatSessionReplicationValve"/>

3. Make the webapp distributable by defining the property "<distributable/>" in web.xml. Then, deploy it in the WSO2 Application Server.

The above steps should also be followed for other WSO2 Application Server nodes in the cluster for which session replication should be enabled. These steps are the standard steps for session replication in a standalone Tomcat cluster environment. After following the above steps, the webapps which are marked as distributable, will have the same session replicated in all cluster nodes.

Conclusion

Embedding Apache Tomcat into the application environment is an easy task as it only requires to call some minimal set of API methods. Even though embedded Tomcat offers the same functionality as a standard Tomcat instance, there are some limitations. This article took you through how we overcame those limitations and how the useful features in Apache Tomcat is also made available in WSO2 Application Server.

References

[1] https://tomcat.apache.org/tomcat-7.0-doc/config/index.html
[2] https://tomcat.apache.org/tomcat-7.0-doc/config/context.html
[3] https://kishanthan.wordpress.com/2012/05/31/how-to-register-jndi-resources-for-webapps-in-wso2-application-server/
[4] https://tomcat.apache.org/tomcat-7.0-doc/appdev/deployment.html#Web_Application_Deployment_Descriptor
[5] https://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html
[6] https://tomcat.apache.org/tomcat-6.0-doc/tribes/introduction.html
[7] https://axis.apache.org/axis2/java/core/docs/clustering-guide.html

Author

Kishanthan Thangarajah, Software Engineer, WSO2 Inc.

 

About Author

  • Kishanthan Thangarajah
  • Senior Technical Lead
  • WSO2