[Tutorial] Enabling SASL Authentication between WSO2 Message Broker and Zookeeper Cluster

  • By Ishara Premadasa
  • 9 May, 2014

Applies to

WSO2 Message Broker 2.2.0
Zookeeper Profile in WSO2 MB 3.4.5

Table of contents

  1. Why security is needed for Message Broker - Zookeeper coordination
  2. What is SASL, DIGEST-MD5 and JAAS?
  3. How SASL authentication happens between the Message Broker and Zookeeper profile
  4. Configuring Zookeeper server for SASL authentication
  5. Configuring WSO2 Message Broker node for SASL authentication
  6. Starting the Zookeeper server profile and Message Broker
  7. Conclusion
  8. References

Why security is needed for Message Broker - Zookeeper coordination

The coordination server is used to propagate queue/topic subscription information, joined/left broker node details, etc. among the Message Broker instances that are connected to the cluster. The recommendation for a typical production MB environment is to use an external Zookeeper three node cluster for coordination. However, from WSO2 MB 2.2.0 onwards, coordination server support is provided starting WSO2 MB instance as a Zookeeper profile, where it is not needed to download Zookeeper from any external site. More details about clustering Message Broker using Zookeeper and Cassandra profiles can be found in the WSO2 Carbon 4.2.0 clustering guide here. [1]

As coordination server cluster is the main component that holds and synchronizes subscription details and broker nodes information, it is very important to prevent any unauthenticated client connecting to the coordination cluster in case some unauthorized client tried to connect a Message Broker instance to the cluster and retrieve coordination information. Therefore, to prevent this, it is necessary to enable a client-server authentication mechanism when connecting a new Message Broker instance (client) to the Zookeeper profile cluster (server).

Apache Zookeeper uses SASL as the default authentication mechanism [2].The default zookeeper server comes with Kerberos (GSSAPI) and DIGEST_MD5 based authentication schemes; as SASL supports pluggable authentication schemes, the user can register a new authentication mechanism and use it for SASL authentication with Zookeeper.

What is SASL, DIGEST-MD5 and JAAS?

Let’s first go through the basic terms that will be used in this article.

SASL (Simple Authentication and Security Layer)

SASL, Simple Authentication and Security Layer is a framework for adding authentication support to connection-based protocols [3]. It allows the user to select different mechanisms to authenticate the client. SASL provides a framework that enables application protocols to negotiate an agreed authentication mechanism. How SASL works is the server provides a list of supported authentication mechanisms (e.g. PLAIN, DIGEST-MD5, etc.) and then the client selects one of those mechanisms and defines it in the client’s login configuration so the server can authenticate the client against that mechanism. There are no additional dependencies required to use SASL with Java as it is a part of the Java Standard Edition.

DIGEST_MD5 authentication mechanism

As Zookeeper supports DIGEST-MD5 scheme [4] for SASL authentication by default we will be using it in the latter part of this article. It is one of the agreed-upon methods that a server can use to negotiate credentials with a client. In this mechanism, the client sends the server a request for a protected resource and the server then responds to the client demanding its authentication details. The client receives the server’s challenge and sends back required credentials in a hashed manner. The server receives this and performs authentication of the challenge response sent by the client. If the check is successful, the client is authenticated to connect to the server.

Java authentication and authorization service (JAAS)

JAAS is used for authentication of users to reliably and securely determine who's currently executing the Java code and it is also used for authorization of users to ensure they have access control rights (permissions) required to carry out the actions performed. [5]

How SASL authentication happens between the Message Broker and Zookeeper profile

As mentioned previously, in this scenario, WSO2 MB acts as the client where the WSO2 MB instance with the Zookeeper profile acts as the coordination server.

  • If the System Property java.security.auth.login.config is defined for the ZooKeeper profile either with server start commands or by using a java.env file, the Zookeeper server creates a javax.security.SaslServer object by using the authentication information derived from the JAAS configuration file (how to configure this data will be discussed later in this article) when there is an SASL client (WSO2 MB here) being connected.
  • If the server is started with requireClientAuthScheme=sasl property enabled, the Zookeeper server allows a non-authenticated client to connect to the cluster, but will not allow to perform any operation other than ping, create session, close session, or sasl-authenticate actions. Therefore, for enhanced security, it is advised to enable this property in zoo.cfg file of the Zookeeper profile.
  • When the Message Broker (client) is started, it first checks whether SASL is enabled for the client. If so, a SaslClient object is initialed and the client goes from CONNECTING to SASL_INITIAL. At this state, the client checks whether it should send an initial response (based on DIGEST-MD5 mechanism). If it should send an initial response, it creates the initial token and sends it to the Zookeeper server and goes to SASL state. The initial token does not contain any data on login credentials and is only an empty byte array.
  • If SASL is not configured on the client, then the client simply goes from SASL_INITIAL to CONNECTED state. This allows non-SASL authenticated Zookeeper clients to interact without modification with a SASL-configured Zookeeper Quorum.
  • At the SASL state, when a client sends the initial token, the Zookeeper server examines it and sends a response to the client, which is stated as a ‘challenge’. The challenge contains data that demands authentication details from the client.
  • When the Message Broker client receives the response, it evaluates challenge and hashes the login data and other required information, and then sends it back to the server.
  • The Zookeeper server again verifies the SASL packet, and authenticates the client’s credentials against the defined username and passwords at server side. If authentication is successful, the Zookeeper server notifies the Broker that it is authentic to connect and it goes to CONNECTED state. If the authentication fails, the Zookeeper goes to AUTH_FAILED state.
  • If the Message Broker node fails at SASL authentication, its connection with the coordination server will be immediately disconnected. If it is needed to keep the broker connection regardless of authentication failure, we need to set system property zookeeper.maintain_connection_despite_sasl_failure=yes at the time of starting the Zookeeper server profile.
  • If the Message Broker client connects to a Zookeeper profile cluster for authentication, the zookeeper server that performs the authentication is decided and selected by the Zookeeper cluster itself. The client is not able to state which coordination node it likes to get connected for authentication.

The following diagram demonstrates the above process between the MB cluster and the Zookeeper profile cluster.

Now that we have gone through the scenario as well as understood the basic terminology that will be used in the article, let’s see how the client and server can be configured to use SASL.

Configuring Zookeeper server for SASL authentication

The Zookeeper server needs three conf files for this named, zoo.cfg, jaas.conf and java.env files. All three files have been created already and can be found in the [MB_ZK_PROFILE_HOME]/repository/conf/etc/ directory by default. First open the zoo.cfg file.

Add the following entries to zoo.cfg file in order to enable SASL.

# renew server-side ticket once an hour. 1000*60*60 = 3600000 milliseconds

At startup, the ZooKeeper server will look at ‘authProvider.x’ entries and interpret the value of those properties as the class name of an authentication plugin. If there are more than one authentication providers, they can be listed here by making the suffix on the property as unique as authProvider.1, authProvider.2, etc. In addition, the SASL provider can also be set using -Dzookeeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider system property.

Open the jaas.conf file in [MB_ZK_PROFILE_HOME]/repository/conf/etc/ directory. You can find the following default entries there.

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required

The ‘Server’ header is always required. ‘org.apache.zookeeper.server.auth.DigestLoginModule required’ entry defines that we are going to use DIGEST-MD5 to authenticate the clients listed under the ‘Server’ entry. Thereafter, it is required to list all the usernames and passwords of authenticated clients by adding ‘user_’ prefix like given below.


Since the Message Broker client uses default admin credentials to connect with Zookeeper, we have given ‘user_admin=admin’ here. If there are new MB users who need to be authenticated, they should be added in jaas.conf file, e.g. if there is a new user named ‘bob’ the modified jaas.conf entry is like,

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required

Open java.env file in [MB_ZK_PROFILE_HOME]/repository/conf/etc/ directory and you will find the below configuration there. The [MB_ZK_PROFILE_HOME] path will be set when you start the MB with Zookeeper profile. Hence, there is no need to explicitly set the path in the given place holder.


At server startup, it will find the jass.conf by taking this path. If the java.env file doesn’t exist, Zookeeper will not enable SASL and start the non-SASL server only.

Configuring WSO2 Message Broker node for SASL authentication

The Message Broker client needs two conf files for SASL, namely jaas.conf and java.env files. These files should be located at [MB_HOME]/repository/conf/security directory. First open the jaas.conf file.

The default entries that can be seen in this file is as below.

Client {
       org.apache.zookeeper.server.auth.DigestLoginModule required

The ‘Client’ header is always required. ‘org.apache.zookeeper.server.auth.DigestLoginModule required’ entry informs the zookeeper server to use DIGEST-MD5 mechanism for this client. Thereafter, we have to specify the username and password of the client. Here, we use default admin user credentials of WSO2 MB to connect with Zookeeper, e.g. if it is for user ‘bob’, modify the jaas.conf as below.

Client {
       org.apache.zookeeper.server.auth.DigestLoginModule required

However, we need to ensure that this user is pre-defined in the jaas.conf file of the Zookeeper server. Otherwise, authentication will fail with the following log at server side.

[2014-05-04 13:16:49,362]  WARN {org.apache.zookeeper.server.auth.SaslServerCallbackHandler} -  User 'bob' not found in list of DIGEST-MD5 authenticateable users.
[2014-05-04 13:16:49,362]  WARN {org.apache.zookeeper.server.auth.SaslServerCallbackHandler} -  No password found for user: null
[2014-05-04 13:16:49,362]  WARN {org.apache.zookeeper.server.ZooKeeperServer} -  Client failed to SASL authenticate: javax.security.sasl.SaslException: DIGEST-MD5: cannot acquire password for bob in realm : zk-sasl-md5
[2014-05-04 13:16:49,363]  WARN {org.apache.zookeeper.server.ZooKeeperServer} -  Closing client connection due to SASL authentication failure.

Open the java.env file or create one in [MB_HOME]/repository/conf/security directory if it doesn’t exist. Add the following entry there.


Replace with path to the Message Broker. At server startup it will find the jass.conf by taking this path. If the jaas.conf file doesn’t exist Message Broker will connect to Zookeeper in non-sasl with following WARN log.

[2014-05-04 13:21:54,865]  WARN {org.apache.zookeeper.ClientCnxn} -  SASL configuration failed: 
javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/home/isha/mb_cluster/mb-1/repository/conf/security/jaas.conf' 
because of a SecurityException: java.lang.SecurityException: /home/isha/mb_cluster/mb-1/repository/conf/security/jaas.conf (No such file or directory) 
Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

Save all the configuration files.

Starting the Zookeeper server profile and Message Broker

Now, start the Zookeeper profile first by using the following command. We have also enabled DEBUG logs for some of the ‘org.apache.zookeeper.x’ classes in order to monitor the authentication process.

sh wso2server.sh -Dprofile=zookeeper

Thereafter, start the Message Broker node. When the client commences the SASL authentication process, the following logs will be displayed at the client side.

[2014-05-04 13:08:58,365] DEBUG {org.apache.zookeeper.client.ZooKeeperSaslClient} -  JAAS loginContext is: Client
[2014-05-04 13:08:58,369]  INFO {org.apache.zookeeper.Login} -  successfully logged in.
[2014-05-04 13:08:58,371]  INFO {org.apache.zookeeper.client.ZooKeeperSaslClient} -  Client will use DIGEST-MD5 as SASL mechanism.
[2014-05-04 13:08:58,386]  INFO {org.apache.zookeeper.ClientCnxn} -  Opening socket connection to server localhost/ Will attempt to SASL-authenticate using Login Context section 'Client'
[2014-05-04 13:08:58,402]  INFO {org.apache.zookeeper.ClientCnxn} -  Socket connection established to localhost/, initiating session
[2014-05-04 13:08:58,408] DEBUG {org.apache.zookeeper.ClientCnxn} -  Session establishment request sent on localhost/
[2014-05-04 13:08:58,459]  INFO {org.apache.zookeeper.ClientCnxn} -  Session establishment complete on server localhost/, sessionid = 0x145c62b199c0000, negotiated timeout = 40000
[2014-05-04 13:08:58,461] DEBUG {org.apache.zookeeper.client.ZooKeeperSaslClient} -  ClientCnxn:sendSaslPacket:length=0
[2014-05-04 13:08:58,481] DEBUG {org.apache.zookeeper.client.ZooKeeperSaslClient} -  saslClient.evaluateChallenge(len=101)
[2014-05-04 13:08:58,482] DEBUG {org.apache.zookeeper.client.ZooKeeperSaslClient} -  ClientCnxn:sendSaslPacket:length=276
[2014-05-04 13:08:58,511] DEBUG {org.apache.zookeeper.client.ZooKeeperSaslClient} -  saslClient.evaluateChallenge(len=40)

Meanwhile Zookeeper server will now try to authenticate Message Broker and the result will be displayed at server side as shown in below logs.

2014-05-04 13:08:58,458]  INFO {org.apache.zookeeper.server.ZooKeeperServer} -  Established session 0x145c62b199c0000 with negotiated timeout 40000 for client /
[2014-05-04 13:08:58,463] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Responding to client SASL token.
[2014-05-04 13:08:58,463] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Size of client SASL token: 0
[2014-05-04 13:08:58,464] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Size of server SASL response: 101
[2014-05-04 13:08:58,482] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Responding to client SASL token.
[2014-05-04 13:08:58,482] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Size of client SASL token: 276
[2014-05-04 13:08:58,484]  INFO {org.apache.zookeeper.server.auth.SaslServerCallbackHandler} -  Successfully authenticated client: authenticationID=admin;  authorizationID=admin.
[2014-05-04 13:08:58,510]  INFO {org.apache.zookeeper.server.auth.SaslServerCallbackHandler} -  Setting authorizedID: admin
[2014-05-04 13:08:58,510]  INFO {org.apache.zookeeper.server.ZooKeeperServer} -  adding SASL authorization for authorizationID: admin
[2014-05-04 13:08:58,511] DEBUG {org.apache.zookeeper.server.ZooKeeperServer} -  Size of server SASL response: 40

Now we have successfully completed the SASL authentication between WSO2 Message Broker and Zookeeper cluster.


WSO2 MB use Zookeeper profiles from MB 2.2.0 onwards in order to support coordination between Message Broker nodes in a cluster. Therefore, it is necessary to prevent any unauthenticated client connecting to the coordination cluster and retrieve coordination information. As Apache Zookeeper uses SASL as its default authentication mechanism, in this article, we discussed the method to configure SASL authentication between the Message Broker and the Zookeeper Server to successfully communicate through SASL.


[1] https://docs.wso2.org/display/CLUSTER420/Clustering+Message+Broker

[2] https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL

[3] https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml

[4] https://en.wikipedia.org/wiki/Digest_access_authentication

[5] https://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tut...