If you are using Registry separation implementation and sharing configuration and governance registries among cluster nodes of any WSO2 product, this document will be useful. Normally, this is an issue users encounter during cluster restart after applying some change to the cluster. As an example, you might have done some change in product instances like applying a patch or any configuration change and then tried restarting the cluster nodes and the central Governance Registry.
During the restart process, if you see that all the nodes are hangs with the below logs, please follow the given steps to fix it.
[2010-12-07 16:42:03,778] INFO - Initializing system...
[2010-12-07 16:42:09,659] INFO - Starting WSO2 Carbon...
[2010-12-07 16:42:09,660] INFO - Operating System : Linux 2.6.18-194.11.1.el5, amd64
[2010-12-07 16:42:09,660] INFO - Java Home : /usr/lib/jvm/java-1.6.0-sun-22.214.171.124.x86_64/jre
[2010-12-07 16:42:09,660] INFO - Java Version : 1.6.0_07
[2010-12-07 16:42:09,660] INFO - Java VM : Java HotSpot(TM) 64-Bit Server VM 10.0-b23,Sun Microsystems Inc.
[2010-12-07 16:42:09,660] INFO - Carbon Home : /home/kindrasi/wso2/wso2greg-3.5.1
[2010-12-07 16:42:09,660] INFO - Java Temp Dir : /home/kindrasi/wso2/wso2greg-3.5.1/tmp
[2010-12-07 16:42:09,661] INFO - User : root, en-US, America/New_York
So, obviously, users will highly doubt this problem causes the change they have made to the cluster. But before investigating based on that assumption, we recommend users do the following check.
Please access the database of the central WSO2 GReg database and select the values of the table REG_CLUSTER_LOCK. If you see the value of REG_LOCK_STATUS as LOCKED, we recommend you set the value to UNLOCKED and try to start the instances starting from the central Governance Registry instance.
This is due to an issue in products released based on Carbon 3.0.x. Root cause of the issue is setting the state LOCKED in the database table REG_CLUSTER_LOCK in Governance Registry database, during the restart process of cluster nodes. Since the products in the cluster is having mounts with central Governance Registry database, the start up of all the cluster nodes hangs.
From this test you can isolate the issue, if you still cannot start, this might be due to the change you have done to the cluster before restarting.
Senior Software Engineer; WSO2 Inc.