Introduction

LoadBalancer

Load balancing is a key concern in a Platform-as-a-Service (PaaS) or a middleware platform deployed on the Cloud. The primary functionality of a load balancer is to distribute the load of the incoming traffic amongst a set of back end worker nodes. This set of worker nodes can be either statically configured or can be dynamically discovered. In WSO2 carbon based products, we use cluster messages which based on axis2 clustering to identify the node join to cluster or leave cluster.

Multi-tenancy - Tenant-aware Load Balancing

If a Cloud deployment has to be able to scale to thousands, hundreds of thousands or millions of tenants, we need tenant-partitioning. This means, for a single Cloud service, there will be multiple clusters, and each of these Service clusters will handle a subset of the tenants in the system. Creating dynamic tenant clusters & tenant partitioning strategies are some of the ongoing research areas. In such a tenant-partitioned deployment, the load balancers themselves need to be tenant-aware, in order to be able to route the requests to the proper tenant clusters. This means the load balancer has to be tenant-aware as well as Service-aware, since it is the Service clusters that are partitioned according to the tenants. WSO2 Load Balancer is now capable of routing messages in both tenant aware and service aware way.

How WSO2 Load Balancer Works

Load Balancer's task is to distribute the load of the incoming traffic amongst a set of back end worker nodes.

We will see how this works now.

when the request comes in to Load Balancer, we have to identify host header and determine cluster domain

Eg: Host header = https://appserver.cloud-test.wso2.com/t/ttt.ttt/carbon/admin/login.jsp

Service domain is appserver domain Then determine tenant Id from the url. Here tenant domain is ttt.ttt so we can determine tenant domain. These parameters can be extracted from host header.

Now we have both service domain and tenant domain so with both of them we can direct this request to correct cluster as we defined in loadbalancer.conf. There we can configure services with their cluster domains and tenant ranges. When load balancer loads, it picks those ranges and cluster domains. See the following sample configuration . You can find those configurations under the “repository/conf” folder. There you will find loadbalancer.conf file.

Here is the configuration for one service with two clustering domains

appserver {
# multiple hosts should be separated by a comma.
hosts appserver.cloud-test.wso2.com;

domains {
carbon.as1.domain {
tenant_range 1-100;
}
carbon.as.domain {
tenant_range *;
}
}
}

Let’s examine this configuration.

Above configuration is for Application service with two service clusters. Let’s examine it line by line. First line says appserver is the main node of this configurations. In next line, you will find hosts entries. There, we have to add host address that maps to application server service. If you need to add multiple service domains, add it with separated by commas.

Next you will find domains; there you will find two domains with predefined tenant ranges. carbon.as1.domain named cluster is responsible for loading tenants with tenant id 1 to 100. carbon.as.domain named cluster is responsible for loading all tenants except the tenants belong to above range.

Then it can forward request to correct cluster based on service domain and tenant domain.

See following diagram to see how request is forwarded to correct cluster.

WSO2 Load balancer supports auto scaling as well. But for this phase of article, we will skip the subject . You can disable auto scaling by setting enable_autoscaler parameter as false in load balancer section of conf file. See following configuration.

# whether autoscaling enable or not
enable_autoscaler false;

How Load Balance End Point works

For load balancing, we are using synapse end point as an end point to determine how messages should be forwarded. So, here we have defined external end point, which is named as Tenant Aware Load Balance End point. It has logic to forward messages to correct back end node. We can define load balance end point in synapse main sequence.

In order to do this, you have to change main sequence with correct load balance end point. If you need to implement your own custom load balancing implementation, you can write your own end point and configure load balancer to use it. As an example, we can load balance based on any other parameter which is available in request header (something similar to tenant domain). It's simple and straightforward. See the above diagram to get clear idea.

You have to add entry to main.xml (available in repository/deployment/server/synapse-configs/default/sequences folder) to use tenant aware load balance endpoint or any other custom endpoint. Add following entry inside send tag.

 org.apache.synapse.endpoints.algorithms.RoundRobin
$system:loadbalancer.conf
true

Advantages and other features of WSO2 Tenant Aware Load Balancer

(1) Tenants are loaded in demand to the predefined cluster based on the configurations in loadbalancer.conf.

(2) Unload unused tenants after particular predefined time.

(3) Tenant Partitioning.

(4) Tenant may have multiple service clusters.

(5) When a new instance is added, notify the Load Balancer instances.

(6) Auto scaling - starting and terminating of instances based on the load. This supports any type of infrastructure as a service. What we have to do is, write simple adapter to auto-scaler service based on the service vendor's API. It’s straight forward and simple. External component's auto-scaler service can add, assign nodes into LB. So Load Balancer is independent from the infrastructure that we are using. This functionality is not available at the moment with Load Balancer and it will be available soon after the next release.

(7) Group communication between clusters using cluster messages.

(8) Notifying all the Load Balancers when a new node is added to cluster.

(9) Data partitioning(data about tenants) - we can use this design to partition data based on the Geography or any other factors. As an example, if we decide that tenant 1 to 100 should load and store their data on United States, we can deploy new application cluster at United States and inform it to Load Balancer. So we have to add configuration to loadbalancer.conf file saying tenant 1 to 100 are loaded at particular cluster(named as wso2.as.usa.domain). Then all requests come to the tenant 1 to 100 will be forwarded to that cluster. So all data related to those tenants (e.g. application files, databases) will be stored at United States servers.

(10) Private jet mode - We can assign some particular application cluster to specific tenant. It is like private jet. If you are willing to pay for entire cluster, we can assign entire cluster for your tenant. This is again can be done using configurations at loadbalancer.conf file. So all requests come to that tenant, will be forwarded to that particular cluster.

(11) Always running an additional cluster(Default cluster) to quickly load new tenant. This cluster is responsible for all newly created tenants in the system and super admin tenant. We can define this cluster in load balancer configuration file with * mark as upper tenant ID.

Conclusion

The WSO2 Tenant Aware Load Balancer is a Load Balancer based on Apache Synapse & WSO2 Carbon. With WSO2 Tenant Aware Load Balancer, we will be able to load balance between multiple service domains with multiple clusters. This new Load Balancer supports tenant partitioning, data partitioning based on tenant partitioning, and private jet mode for special tenants. Within each cluster, Load Balancer will balance load between nodes in round robin fashion. This is the easiest way to balance the load between WSO2 Carbon based servers in a tenant aware way.

References

Home page of Load Balancer

The role of a Load Balancer in a Platform-as-a-Servicer

Author

Sanjeewa Malalgoda, Software Engineer, WSO2, sanjeewa AT wso2 DOT com