×
2013/10/17
17 Oct, 2013

Clustering WSO2 Identity Server and Load balancing it with WSO2 Elastic Load Balancer

  • Chamara Ariyarathne
  • Associate Technical Lead - QA - WSO2

The WSO2 Identity server 4.5.0 and WSO2 Enterprise Load Balancer 2.1.0 is now released. So with this post I'm going to guide you with each and every and complete set of steps on how to cluster Identity server with two nodes and loadbalance it with ELB. And this setup is going to be in one Ubuntu machine.

Prerequisites:

1. Ubuntu 12.04 or a Linux similar platform

2. Java 1.6 or above (I'm using 1.7.0_25 64-bit)

3. Mysql (I'm using Ver 14.14 Distrib 5.5.32)

Let's go through the steps on how to cluster IS

1. Pointing to same User store and User Management db.

Being a cluster of IS, every node need to share the same User Store and User Management databases. So that every request come through load balancer will be served by any node similarly.

For User Store you can use any LDAP as Active Directory, Apache DS, OpenLDAP etc. Here I'm using the JDBC userstore which is supported by Identity Server to use a relational database as the user store.

I'm using mysql database for this.

Create a new database

mysql -uroot -proot  
 mysql> create database is_user_store;

Populate the tables needed. For that you can use the db scripts in the dbscripts directory in Identity Server home.

${IS_HOME}/dbscripts/mysql.sql  
 ${IS_HOME}/dbscripts/identity/mysql.sql  
 ${IS_HOME}/dbscripts/service-provider/mysql.sql  

 mysql> source mysql.sql;  
 mysql> source identity/mysql.sql;  
 mysql> source service-provider/mysql.sql;

Now you have to use this database in the configurations. For the User Store you have to add the following configurations;

In the;

${IS_HOME}/repository/conf/datasources/master-datasources.xml

Have the configuration


       WSO2_USERSTORE_DB  
       The datasource used for registry and user manager  
         
         jdbc/WSO2UserStoreDB  
         
         
           
           jdbc:mysql://localhost:3306/is_user_store  
           root  
           root  
           com.mysql.jdbc.Driver  
           50  
           60000  
           true  
           SELECT 1  
           30000

And that jndi database reference have to be used in the user-mgt.xml as to use it as the userstore

${IS_HOME}/repository/conf/user-mgt.xml
  
       org.wso2.carbon.user.core.tenant.JDBCTenantManager  
       false  
       100  
       false  
       default  
       SHA-256  
       true  
       true  
       true  
       false  
       ^[\S]{5,30}$  
       ^[\S]{5,30}$  
       ^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$  
       ^[\S]{3,30}$  
       ^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$  
       ^[\S]{3,30}$  
       true  
       100  
       100  
       false  
       false  
       jdbc/WSO2UserStoreDB

And for that to be the Primary userstore of the IS that need to be on top in the user-mgt.xml as the first mentioned userstore configuration. Every other userstore configuration you specify will be secondary userstores with a domain name.

and this has to be done in both of the IS nodes.

Now lets create another db and set it as the User Management db. Run the scripts to populate tables and refer it in the master-datasources.xml as following.

  
       WSO2_UM_DB  
       The datasource used for registry and user manager  
         
         jdbc/WSO2UserMgtDB  
         
         
           
           jdbc:mysql://localhost:3306/is_user_mgt  
           root  
           root  
           com.mysql.jdbc.Driver  
           50  
           60000  
           true  
           SELECT 1  
           30000

You have to refer this in the user-mgt.xml as in following.

  
         true  
         admin  
           
            admin  
            admin  
           
       everyone   
       jdbc/WSO2UserMgtDB

Now the User management related configurations are done. The next step is sharing the config and governance registries among each IS nodes. That is called Registry Mounting.

2. Registry Mounting

Let's create a database to be used in the mount configurations so that it will be the same repository for both IS nodes to share as the Config and Governance collections. That will be called the Mounted db. However for each IS node, the local registry will still be pointed to each and every node's H2 database. That configuration is already there in the master-datasources.xml. That need not to be changed.

You'll have to create a database in mysql and populate the tables. Let's see how it will be refered in the configurations.

First in the; 

${IS_HOME}/repository/conf/datasources/master-datasources.xml 
  
       WSO2_MOUNT_DB  
       The datasource used for registry and user manager  
         
         jdbc/WSO2ISMountDB  
         
         
           
           jdbc:mysql://localhost:3306/is_mount_db  
           root  
           root  
           com.mysql.jdbc.Driver  
           50  
           60000  
           true  
           SELECT 1  
           30000

And now this jndi database reference will be used in both nodes as in; 

${IS_HOME}/repository/conf/registry.xml
wso2registry  
   false  
   true  
   /  
     
     jdbc/WSO2CarbonDB  
     
     
     jdbc/WSO2ISMountDB  
     
     
       
       application/xacml-policy+xml  
       
     

  
     instanceid  
     mounted_registry  
     false  
     true  
     /   
     
     
     instanceid  
     /_system/nodes  
     
     
     instanceid  
     /_system/governance

Now let's look at the clustering configurations.

3. Clustering of IS nodes

We need to cluster two IS nodes as management nodes, and load balance it with WSO2 ELB.

First let's look at Identity Server configurations.

This is for the IS node 1.

${IS_HOME}/repository/conf/axis2/axis2.xml

There is a section as "clustering". You have to find the place and edit as following.

 
wka

We have to specify a domain for Identity Server cluster. And this domain name need to be unique from other WSO2 product clusters. Even from ELB domain.

 wso2.is.chamara.domain  
 4001  
  
         
         
         
   

${IS_HOME}/repository/conf/carbon.xml
 1
 wso2.is.chamara.com  
 wso2.is.chamara.com

This hostname will be the one that is used by IS cluster and the ELB. It need to be specified in the;

 /etc/hosts

as;

 127.0.0.1 wso2.is.chamara.com

In the carbon.xml we are specifying the deployment synchronizer configuration as well. Here I'm using svn based deployment synchronizer. So you'll have to find your own svn server and specify a location. Or else you can use Registry Based Deployment Synchronizer

  
     true  
     true  
     true  
     svn  
     https://svnexample.wso2.com/svn/chamara450  
     wso2  
     wso2123  
     true

I'm going to run ELB on default HTTP/HTTPS ports which are 80/443.

So we have to specify those as proxy ports in the

${IS_HOME}/repository/conf/tomcat/catalina-server.xml

Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9763"  
         proxyPort="80"

Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9443"  
         proxyPort="443"

In the IS node 2 the configurations are almost same as for node1. The only differences that need to be changed are; in;

${IS_HOME}/repository/conf/axis2/axis2.xml
4002

and in;

${IS_HOME}/repository/conf/carbon.xml
 2

4. Sharing of same keystores in IS cluster and ELB

In this step I'm going to explain how to use the same keystore in three servers and having certificates in the truststores. We can always use the default keystore wso2carbon.jks and it's public certificates. But in this case since we are using ELB in default ports, the servers need to have same keystore with the Common Name as the domain name (CN=wso2.is.chamara.com). So here are the steps to generate your own keystore.

$ keytool -genkey -alias iscarbon -keyalg RSA -keysize 1024 -keypass iscarbon -keystore iscarbon.jks -storepass iscarbon  
 What is your first and last name?  
  [Unknown]: wso2.is.chamara.com  
 What is the name of your organizational unit?  
  [Unknown]: users.wso2  
 What is the name of your organization?  
  [Unknown]: wso2  
 What is the name of your City or Locality?  
  [Unknown]: Colombo  
 What is the name of your State or Province?  
  [Unknown]: Western  
 What is the two-letter country code for this unit?  
  [Unknown]: 94  
 Is CN=wso2.is.chamara.com, OU=users.wso2, O=wso2, L=Colombo, ST=Western, C=94 correct?  
  [no]: yes

Here for local testing purposes it is ok to use self signed certificate. If this is going to be used in production the public certificate need to be signed by Certificate Authority.

Now export the public certificate and import it to the server trust store. Here it need to be imported in two IS nodes and the ELB.

$ keytool -export -alias iscarbon -keystore iscarbon.jks -storepass iscarbon -file iscarbon.pem  
$ keytool -import -alias iscarbon -file iscarbon.pem -keystore client-truststore.jks -storepass wso2carbon

Now we need to change the configuration files to use this new keystore as the server keystore. For that following locations needs to be updated accordingly. 

identity.xml: ${carbon.home}/repository/resources/security/iscarbon.jks  
 identity.xml: iscarbon  
 security/cipher-text.properties: Carbon.Security.KeyStore.Password=[iscarbon]  
 security/cipher-text.properties: Carbon.Security.KeyStore.KeyPassword=[iscarbon]  
 carbon.xml: ${carbon.home}/repository/resources/security/iscarbon.jks  
 carbon.xml: iscarbon  
 carbon.xml: iscarbon  
 carbon.xml: iscarbon  
 axis2/axis2.xml: repository/resources/security/iscarbon.jks  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: repository/resources/security/iscarbon.jks  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon  
 axis2/axis2.xml: iscarbon

5. Configuring the Elastic Load Balancer

We are done with configuring the IS nodes. Lets look at how to configure the ELB

${ELB_HOME}/repository/conf/axis2/axis2.xml
  
 wso2.chamara.lb.domain

this is a individual port unique from is nodes' localMemberPort

 4100

For changing Transport ports (Proxy ports in IS nodes)

  
    80  
  
     443

In the file

${ELB_HOME}/repository/conf/loadbalancer.conf

Remove all the services entries and add the following.

identity {  
     domains  {  
       wso2.is.chamara.domain {  
         tenant_range  *;  
         group_mgt_port 4000;  
         mgt {  
           hosts wso2.is.chamara.com;  
         }  
       }  
     }  
   }

In the ELB also you need to specify keystore configurations as above, in order to use the iscarbon.jks. Here you don't have add configure or add any identity.xml.

6. Running the cluster

Now we have come to the last part. First run the ELB. For run that you need to be the superuser of that computer since we are running ELB on default ports.

{ELB_HOME}# ./bin/wso2server.sh

Then run two IS servers.

{IS_HOME}$ ./bin/wso2server.sh

You will see the following logs in ELB while IS nodes joining ELB.

[2013-09-04 08:21:15,211] INFO - HazelcastGroupManagementAgent Member joined [ad2e310c-bf27-4bf3-9e03-afbe2117b6dd]: /127.0.0.1:4001  
 [2013-09-04 08:21:18,275] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:21:18,276] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster  

[2013-09-04 08:24:59,428] INFO - HazelcastGroupManagementAgent Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:02,493] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:02,494] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster

And also while IS node joins each other following log will be appeared.

Node 1

[2013-09-04 08:22:15,083] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:22:23,314] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8228ms  
 [2013-09-04 08:22:23,365] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [bb0a0482-aea0-4895-a2f1-68ee721a38a5] - Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:22:23,374] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

[2013-09-04 08:24:59,430] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:01,494] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true

Node 2

[2013-09-04 08:24:53,226] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:25:01,451] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8223ms  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [3db96e0f-d4e9-4c45-bbdf-6331682a61cd] - Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,492] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true

Now you can access the IS cluster Management Consoles using following url;

https://wso2.is.chamara.com/carbon/

Login using the default admin user {admin:admin}

Source : https://chamaraanuei.blogspot.com/2013/09/clustering-wso2-identity-server-and.html

 

About Author

  • Chamara Ariyarathne
  • Associate Technical Lead - QA
  • WSO2

API Management
Integration
IAM
Internal Developer Platform
Solutions
Resources
Support
Company
Sign Up For Our Newsletter

Follow us

Learn About Security At WSO2 →