Clustering WSO2 Identity Server and Load balancing it with WSO2 Elastic Load Balancer

  • By Chamara Ariyarathne
  • 17 Oct, 2013

The WSO2 Identity server 4.5.0 and WSO2 Enterprise Load Balancer 2.1.0 is now released. So with this post I'm going to guide you with each and every and complete set of steps on how to cluster Identity server with two nodes and loadbalance it with ELB. And this setup is going to be in one Ubuntu machine.

Prerequisites:

1. Ubuntu 12.04 or a Linux similar platform

2. Java 1.6 or above (I'm using 1.7.0_25 64-bit)

3. Mysql (I'm using Ver 14.14 Distrib 5.5.32)

Let's go through the steps on how to cluster IS

1. Pointing to same User store and User Management db.

Being a cluster of IS, every node need to share the same User Store and User Management databases. So that every request come through load balancer will be served by any node similarly.

For User Store you can use any LDAP as Active Directory, Apache DS, OpenLDAP etc. Here I'm using the JDBC userstore which is supported by Identity Server to use a relational database as the user store.

I'm using mysql database for this.

Create a new database

mysql -uroot -proot  
 mysql> create database is_user_store;  

Populate the tables needed. For that you can use the db scripts in the dbscripts directory in Identity Server home.

${IS_HOME}/dbscripts/mysql.sql  
 ${IS_HOME}/dbscripts/identity/mysql.sql  
 ${IS_HOME}/dbscripts/service-provider/mysql.sql  
 mysql> source mysql.sql;  
 mysql> source identity/mysql.sql;  
 mysql> source service-provider/mysql.sql;  

Now you have to use this database in the configurations. For the User Store you have to add the following configurations;

In the;

${IS_HOME}/repository/conf/datasources/master-datasources.xml  

Have the configuration

WSO2_USERSTORE_DBThe datasource used for registry and user managerjdbc/WSO2UserStoreDBjdbc:mysql://localhost:3306/is_user_storerootrootcom.mysql.jdbc.Driver5060000trueSELECT 130000

And that jndi database reference have to be used in the user-mgt.xml as to use it as the userstore

${IS_HOME}/repository/conf/user-mgt.xml
org.wso2.carbon.user.core.tenant.JDBCTenantManagerfalse100falsedefaultSHA-256truetruetruefalse^[\S]{5,30}$^[\S]{5,30}$^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$^[\S]{3,30}$^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$^[\S]{3,30}$true100100falsefalsejdbc/WSO2UserStoreDB

And for that to be the Primary userstore of the IS that need to be on top in the user-mgt.xml as the first mentioned userstore configuration. Every other userstore configuration you specify will be secondary userstores with a domain name.

and this has to be done in both of the IS nodes.

Now lets create another db and set it as the User Management db. Run the scripts to populate tables and refer it in the master-datasources.xml as following.

WSO2_UM_DBThe datasource used for registry and user managerjdbc/WSO2UserMgtDBjdbc:mysql://localhost:3306/is_user_mgtrootrootcom.mysql.jdbc.Driver5060000trueSELECT 130000

You have to refer this in the user-mgt.xml as in following.

trueadminadminadmineveryonejdbc/WSO2UserMgtDB

Now the User management related configurations are done. The next step is sharing the config and governance registries among each IS nodes. That is called Registry Mounting.

2. Registry Mounting

Let's create a database to be used in the mount configurations so that it will be the same repository for both IS nodes to share as the Config and Governance collections. That will be called the Mounted db. However for each IS node, the local registry will still be pointed to each and every node's H2 database. That configuration is already there in the master-datasources.xml. That need not to be changed.

You'll have to create a database in mysql and populate the tables. Let's see how it will be refered in the configurations.

First in the;

${IS_HOME}/repository/conf/datasources/master-datasources.xml 
WSO2_MOUNT_DBThe datasource used for registry and user managerjdbc/WSO2ISMountDBjdbc:mysql://localhost:3306/is_mount_dbrootrootcom.mysql.jdbc.Driver5060000trueSELECT 130000

And now this jndi database reference will be used in both nodes as in;

${IS_HOME}/repository/conf/registry.xml
wso2registryfalsetrue/jdbc/WSO2CarbonDBjdbc/WSO2ISMountDBapplication/xacml-policy+xml
instanceidmounted_registryfalsetrue/instanceid/_system/nodesinstanceid/_system/governance

Now let's look at the clustering configurations.

3. Clustering of IS nodes

We need to cluster two IS nodes as management nodes, and load balance it with WSO2 ELB.

First let's look at Identity Server configurations.

This is for the IS node 1.

${IS_HOME}/repository/conf/axis2/axis2.xml

There is a section as "clustering". You have to find the place and edit as following.

wka

We have to specify a domain for Identity Server cluster. And this domain name need to be unique from other WSO2 product clusters. Even from ELB domain.

 wso2.is.chamara.domain
 4001
${IS_HOME}/repository/conf/carbon.xml
 1
 wso2.is.chamara.com
 wso2.is.chamara.com

This hostname will be the one that is used by IS cluster and the ELB. It need to be specified in the;

 /etc/hosts  

as;

 127.0.0.1 wso2.is.chamara.com  

In the carbon.xml we are specifying the deployment synchronizer configuration as well. Here I'm using svn based deployment synchronizer. So you'll have to find your own svn server and specify a location. Or else you can use Registry Based Deployment Synchronizer

truetruetruesvnhttps://svnexample.wso2.com/svn/chamara450wso2wso2123true

I'm going to run ELB on default HTTP/HTTPS ports which are 80/443.

So we have to specify those as proxy ports in the

${IS_HOME}/repository/conf/tomcat/catalina-server.xml
Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9763"  
         proxyPort="80"
Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9443"  
         proxyPort="443" 

In the IS node 2 the configurations are almost same as for node1. The only differences that need to be changed are; in;

${IS_HOME}/repository/conf/axis2/axis2.xml
4002

and in;

${IS_HOME}/repository/conf/carbon.xml
 2

4. Sharing of same keystores in IS cluster and ELB

In this step I'm going to explain how to use the same keystore in three servers and having certificates in the truststores. We can always use the default keystore wso2carbon.jks and it's public certificates. But in this case since we are using ELB in default ports, the servers need to have same keystore with the Common Name as the domain name (CN=wso2.is.chamara.com). So here are the steps to generate your own keystore.

$ keytool -genkey -alias iscarbon -keyalg RSA -keysize 1024 -keypass iscarbon -keystore iscarbon.jks -storepass iscarbon  
 What is your first and last name?  
  [Unknown]: wso2.is.chamara.com  
 What is the name of your organizational unit?  
  [Unknown]: users.wso2  
 What is the name of your organization?  
  [Unknown]: wso2  
 What is the name of your City or Locality?  
  [Unknown]: Colombo  
 What is the name of your State or Province?  
  [Unknown]: Western  
 What is the two-letter country code for this unit?  
  [Unknown]: 94  
 Is CN=wso2.is.chamara.com, OU=users.wso2, O=wso2, L=Colombo, ST=Western, C=94 correct?  
  [no]: yes  

Here for local testing purposes it is ok to use self signed certificate. If this is going to be used in production the public certificate need to be signed by Certificate Authority.

Now export the public certificate and import it to the server trust store. Here it need to be imported in two IS nodes and the ELB.

$ keytool -export -alias iscarbon -keystore iscarbon.jks -storepass iscarbon -file iscarbon.pem  
$ keytool -import -alias iscarbon -file iscarbon.pem -keystore client-truststore.jks -storepass wso2carbon  

Now we need to change the configuration files to use this new keystore as the server keystore. For that following locations needs to be updated accordingly.

identity.xml: ${carbon.home}/repository/resources/security/iscarbon.jks  
 identity.xml: iscarbon  
 security/cipher-text.properties: Carbon.Security.KeyStore.Password=[iscarbon]  
 security/cipher-text.properties: Carbon.Security.KeyStore.KeyPassword=[iscarbon]  
 carbon.xml: ${carbon.home}/repository/resources/security/iscarbon.jks  
 carbon.xml: iscarbon  
 carbon.xml: iscarbon  
 carbon.xml: iscarbon  
 axis2/axis2.xml: repository/resources/security/iscarbon.jks  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: repository/resources/security/iscarbon.jks  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon

5. Configuring the Elastic Load Balancer

We are done with configuring the IS nodes. Lets look at how to configure the ELB

${ELB_HOME}/repository/conf/axis2/axis2.xml
 wso2.chamara.lb.domain

this is a individual port unique from is nodes' localMemberPort

 4100

For changing Transport ports (Proxy ports in IS nodes)

80
443

In the file

${ELB_HOME}/repository/conf/loadbalancer.conf

Remove all the services entries and add the following.

identity {  
     domains  {  
       wso2.is.chamara.domain {  
         tenant_range  *;  
         group_mgt_port 4000;  
         mgt {  
           hosts wso2.is.chamara.com;  
         }  
       }  
     }  
   }  

In the ELB also you need to specify keystore configurations as above, in order to use the iscarbon.jks. Here you don't have add configure or add any identity.xml.

6. Running the cluster

Now we have come to the last part. First run the ELB. For run that you need to be the superuser of that computer since we are running ELB on default ports.

{ELB_HOME}# ./bin/wso2server.sh  

Then run two IS servers.

{IS_HOME}$ ./bin/wso2server.sh 

You will see the following logs in ELB while IS nodes joining ELB.

[2013-09-04 08:21:15,211] INFO - HazelcastGroupManagementAgent Member joined [ad2e310c-bf27-4bf3-9e03-afbe2117b6dd]: /127.0.0.1:4001  
 [2013-09-04 08:21:18,275] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:21:18,276] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster  
[2013-09-04 08:24:59,428] INFO - HazelcastGroupManagementAgent Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:02,493] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:02,494] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster  

And also while IS node joins each other following log will be appeared.

Node 1

[2013-09-04 08:22:15,083] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:22:23,314] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8228ms  
 [2013-09-04 08:22:23,365] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [bb0a0482-aea0-4895-a2f1-68ee721a38a5] - Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:22:23,374] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
[2013-09-04 08:24:59,430] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:01,494] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

Node 2

[2013-09-04 08:24:53,226] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:25:01,451] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8223ms  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [3db96e0f-d4e9-4c45-bbdf-6331682a61cd] - Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,492] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

Now you can access the IS cluster Management Consoles using following url;

https://wso2.is.chamara.com/carbon/

Login using the default admin user {admin:admin}

Source : https://chamaraanuei.blogspot.com/2013/09/clustering-wso2-identity-server-and.html