Securely Deploying Tomcat Server in a Read-Only File System in Kubernetes

Apache Tomcat usage and ramifications in WSO2 Asgardeo

Apache Tomcat is a popular application server that hosts Java Servlets and JSPs. Asgardeo uses Apache Tomcat to host its website component, which acts as a web server. 

Initially, the Tomcat server's file system was created with read/write permissions. However, due to Kubernetes' security practices, the file system was reconfigured to have read-only permission to prevent potential security breaches. As a result, the Tomcat server container was also configured as a read-only file system. Having read only permission causes an error as Tomcat fails to publish its own logs to the /logs directory in the file system, as publishing logs is a write operation. Consequently, the deployment failed. This article explains how to resolve this issue.

Deploying a Tomcat server with a read-only file system

When deploying the Tomcat server in a Kubernetes environment, the read-only configuration of the container's file system can cause the deployment to fail. Initially, it will fail when write operations are attempted on the read-only file system. There are two main scenarios where write operations occur in the Tomcat deployment. 

1. Log files are getting published to the container.
2. Copying the WAR file into the Tomcat work directory.

An effective solution is to redirect the logs to the console and permit write operations only in essential directories. To resolve it, we can divide it into three sections:

1. How can Tomcat logs be redirected to the console without writing to a file?
2. Identify the directories where write operations are required and exclude the read-only file system from those directories.
3. Deployment configurations are required to deploy a Tomcat server with a read-only file system.

In Kubernetes environments, security vulnerabilities may occur, and due to these vulnerabilities, there are many security recommendations we need to adapt to fortify our deployments. One of the recommended approaches is to ensure your critical file systems remain under read-only permissions. When you try to deploy a Tomcat server in a Kubernetes environment with a read-only file system configuration for the container, your deployment will fail. The failure occurred during the startup of the Tomcat server as it attempted to write several log files in the /log directory. However, this fails initially as the file system is read-only and cannot support write operations.

First, let's assess how default Tomcat logging is configured. There are five common log types defined in Tomcat. These common logs are listed below:

1. Access logs: Access logs record information about requests made to the server.
2. Catalina logs: Catalina logs consist of comprehensive information about the server's internal operations.
3. Host-manager logs: Host-manager logs record information about the management of virtual hosts on the server.
4. Manager logs: Manager logs gather information about the deployment and management of web applications on the server.
5. Localhost logs: Localhost logs write log messages to a file for a specific virtual host on the server.

When initially deployed to the Tomcat server, the above-mentioned logs will be written into a file under the /logs directory on the Tomcat server.

Tomcat logging is often set up to write log files to the TOMCAT HOME/logs directory. The default Tomcat logging settings are displayed in the code snippet, which is located in the TOMCAT HOME/conf/logging.properties directory. For additional information, see this link.

handlers = 1catalina.org.apache.juli.AsyncFileHandler, 
2localhost.org.apache.juli.AsyncFileHandler, 
3manager.org.apache.juli.AsyncFileHandler,
4host-manager.org.apache.juli.AsyncFileHandler, 
java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.AsyncFileHandler, 
java.util.logging.ConsoleHandler

Tomcat has four main components: Catalina, localhost, manager, and host-manager. They each produce a log file that is stored using the org.apache.juli.AsyncFileHandler handler. For more details, please refer to the documentation.

Apache has implemented its own version of several crucial elements in the java.util.logging API. This implementation allows a property file to be incorporated into more complex structures, which enables greater flexibility in managing logger assignment and handler definition. By default, the log level threshold for a handler is set to INFO, but it can be modified to any of the following values: SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, or ALL. Additionally, you can set a logging level and specify a specific program for log gathering purposes.

How to redirect Tomcat logs to the console

In the logging configuration file, removing the file handlers and keeping only the console handlers will stop publishing the logs to a file, and the following new logging configuration should be considered. 

However, there should be an acceptable approach to redirecting logs to the console in order to archive the logs; thus, the following additional logging settings should be considered. 

handlers = 1catalina.java.util.logging.ConsoleHandler, \
   2localhost.java.util.logging.ConsoleHandler, \
   3manager.java.util.logging.ConsoleHandler, \
   4host-manager.java.util.logging.ConsoleHandler
.handlers = 1catalina.java.util.logging.ConsoleHandler

Observe that the new console handlers are used to redirect Tomcat logs to the console using java.util.logging.ConsoleHandler.

• <<handler-name>.level specifies the default level for the handler (the default is set to Level.INFO).
• <<handler-name>.filter specifies the name of a filter class to use (the default is no filter).
• <<handler-name>.formatter specifies the name of a formatter class to use (default is java.util.logging.SimpleFormatter).
• <<handler-name>.encoding the name of the character set encoding to use (default is set to the default platform encoding).

Consider the following code segment:

4host-manager.java.util.logging.ConsoleHandler.level = INFO
4host-manager.java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
4host-manager.java.util.logging.SimpleFormatter.format= [%1$tc] [:host-manager:] %4$s %3$s %5$s %n 
4host-manager.java.util.logging.ConsoleHandler.encoding = UTF-8

Deployment configurations are required to deploy a Tomcat server with a read-only file system

As previously noted, it's not possible to redirect Tomcat access logs to the console since the access log configurations are set in the server.xml file. In order to create a new access log configuration, a custom valve implementation must be developed. To address this issue, it's essential to attach an emptyDir volume to the /logs directory in Tomcat. By mounting the empty directory volume to /usr/local/tomcat/logs, the Tomcat server can write the access log to a file without encountering any deployment errors.

Before moving on, let’s check what emptyDir is in Kubernetes

An emptyDir is one of the volume types in Kubernetes. It will be created at container startup and when a pod is assigned to a node. The created emptyDir will run until the particular pod runs in the assigned node. The emptyDir volumes are initiated empty, and any containers within the pod can read from and write to them. It is important to note that if the pod containing the emptyDir is deleted from the node, all data stored within the emptyDir will be lost.

Configuring the emptyDir volume mounts to exclude and allow write operations to selected directories

It is best to mount an emptyDir volume to the default logging directory in Tomcat, which is the "/logs" directory. Keep in mind that when you mount an emptyDir volume, it will initially create an empty directory in the respective location and may result in removing your other files in that same directory. Because of emptyDir, all containers in a pod can read and write the same files into the same emptyDir. This approach will aid in resolving our issue by letting the Tomcat write the logs to the emptyDir volume mount, which is mounted to the "/logs" directory.

volumes:            
emptyDir: {}       
- name: tomcat-logs-dir         
Once you define the emptyDir you have to mount it to the tomcat “/logs” directory.
volumeMounts:
- name: tomcat-logs-dir             
mountPath: /usr/local/tomcat/logs

We need to define an empty directory volume mount for the path /usr/local/tomcat/work. This directory is used during runtime to write any necessary files, such as the generated servlet code for JSPs. Since write operations are required for this directory, it is necessary to define it as an empty directory volume to allow write operations when deploying the Tomcat server with a read-only file system.

In Asgardeo, the required WAR file, which contains the web archive, will be copied to the /tomcat/work directory before the main container is deployed using an init container. Since this operation involves writing to the container, write operations must be allowed for this path, you can add an emptyDir mount as mentioned below in your deployment yaml file.

volumes:            
emptyDir: {}       
- name: tomcat-work-dir         
volumeMounts:                    
- name: tomcat-work-dir             
mountPath: /usr/local/tomcat/work

The XML files where server-side configurations are defined will be stored in the path tomcat/conf/Catalina/localhost. Tomcat usually keeps the configuration XML files in this directory, this file is loaded when the server is initiated. When you define a context using an XML file, you need to add the corresponding configuration-based XML file to this directory.

- name: tomcat-localhost-dir
  mountPath: /usr/local/tomcat/conf/Catalina/localhost
- name: tomcat-localhost-dir
  emptyDir: {}

Since there are write operations happening in this directory, you need to provision write permission for this directory. You can also define an empty directory for this folder path and specify write operations.

We have now identified that when you need to deploy the Tomcat server in a read-only file system, there are several file directories that need to be excluded, and we need to allow write operations into those directories. Since these file paths are essential, we need to accept the security risks associated with those directories.

How to access the Tomcat logs

By implementing the read-only file system configuration, you can securely launch Tomcat for an existing Kubernetes deployment. Accessing the logs is easy; simply log into the container and review the console logs, except for access logs, which can be found in the /logs directory. To ensure a secure deployment, it's best to configure the read-only file system for all containers and volumes associated with them and provide a secure storage solution for the log data.

The Azure container apps service offered for Kubernetes deployments enables configuring logs to be forwarded to an Azure log analytics workspace. Subsequently, the logs can be queried from the workspace. For further information, please refer to Monitor Logs in Azure Container Apps with Log Analytics.

We hope that this blog will provide valuable insights and guidance on configuring Apache logs in your deployment. By following the steps outlined, you can enhance the security of your system and ensure that your log data is stored safely.