Usage Metering in a Cloud Environment
WSO2 StratosLive is a public cloud deployment. There needs to be a way of measuring how tenants consume cloud resources. It is necessary to measure them to control or facilitate the resources consumed by the tenants. StratosLive encourages tenants to make full use of it's resources. But there needs to be a control mechanism too. Billing, Metering and Throttling components are responsible for measuring resource usage, charging the tenants and controlling access to resources in StratosLive.
Metering is the base for Billing and Throttling. It measures bandwidth and storage consumed by tenants. Throttling runs its throttling rules against the metered data and takes decisions on access controlling to resources. This resources are, webapps, services, registry storage and registry items. Billing, if enabled (which is enabled in StratosLive), charges customers based on the metered data in a monthly basis. Billing considers not only the metered data, but also the usage plan which the tenant is subscribed and the pricing strategy.
How to Configure Usage in a Cloud Environment Using WSO2 Stratos
Before we begin, let’s understand a few terms.
A component which measures resource usage (use listening technology) and stores and published it to other server to record it. This usage metering process uses some advanced listening technologies to avoid slow down or blocks entire processes.
For example, let say some Web service (hosted by sanjeewa.com) is invoked by some client. This invocation takes 100kb service bandwidth. The usage agent stores this entry and the tenant (sanjeewa.com) gets service bandwidth of 100kb at this time.
In order for usage metering to work properly, we have to do few configurations in Stratos services. Usage metering is based on a simple concept that is listening. There are few listeners to listen to the actions performed by users and those listeners report what they hear to some other server (Business Activity Monitor). Those records will be stored in some ordered format. We will first identify those listeners.
01. Registry Listener – Listens to each and every action related to registry (get, put, add, delete etc.. ) Then stores them as bandwidth usage.
02. User Add Listener – This listener is there to listen to user adding actions and keeps updating the user count of tenants.
03. Service/Web App bandwidth listener – Mainly listens to incoming and out going message sizes and stores those data as bandwidth usages. For this listener, the usage agent uses tomcat valve data statics.
The following diagram depicts an overview of the usage metering process.
Figure : Overview Of Usage Metering Process
Data publisher – Data publisher is available inside the usage agent. When usage agent starts, the Data publisher creates the connection to the end point of BAM. In order to do this, we have to provide the Server URL the BAM instance is running on.
Let’s see how we do this configuration. The usage metering process happens at each and every Stratos service so usage agent is available in every service of Stratos. You can see carbon.xml file in every service inside the repository/conf folder. In carbon.xml file you will see following entry.
You have to give correct BAM server URL with /services part. You can see this URL when BAM server starts up. Only then will usage agent be able to publish data to BAM.
These metered usage data will be available for any decision making process. Users(Tenants) can view their usage and the super administrator can view all tenant usages. Throttling manager uses those records to check whether users exceeded their allowed limits or not.
There are some other important parameters to configure for publisher. Those parameters are used for things like how we accumulate, how often we publish them to BAM etc. Since those configurations are developer-level changes, they are not discussed in this article.
Advanced configuration for data publisher in WSO2 Stratos
Now that we have discussed how to do basic configurations related to a Stratos usage agent, let’s take a look at how to fine-tune parameters.
A usage agent is responsible for data publishing as mentioned before. Normally, this publishing is done after some data accumulation. Based on applied load, we may need to configure publishing. The main reason for this configuration change is because there were unusual CPU usage when we published data and we fixed the number of records and Execution time.
The previous implementation of usage agent runs in 5 min intervals. It causes sudden CPU usage increases due to the large number of records we are summarizing and persisting. To solve this issue, we use a task which runs forever (until server shutdown) and persists a small number of records in short intervals. Number of records and time interval can be configured via a configuration file.
The newly introduced configuration file should be copied into CARBON_HOME/repository/conf/advanced directory and should be named as usage-throttling-agent-config.xml. Default values for configuration options are as follows.
- Startup delay - 60 seconds
- Execution interval - 100 milli seconds
- Records per execution - 100
If the file is not available, usage agent will use the default values which are tested, optimized values for public deployment with 5000 active tenants. You will see the following entries in usage-throttling-agent-config.xml file.
You can change the values and analyze the results. See the following diagram for a better understanding of CPU Usage due to a large number of records. These results may change due to the traffic to your servers and the number of tenants.
Figure : Fixed time Publishing with more than 1 second delay VS Configured publishing with 100 milli second publish time and record limitation is 100
As you can see in this figure you will see significant improvement on CPU usage
We can use usage metering and throttling effectively on cloud environment
We can achieve higher performance by fine tuning usage parameters
Sanjeewa Malalgoda, Software Engineer, WSO2 Inc.