[Article] Enabling Monetization of APIs with WSO2 API Manager

Table of contents

• Prerequisites
• Introduction
• Monetization with WSO2 API Manager
• How subscriptions happen
• Data retrieving model for monetization
• Controlling non-billing developer
• Controlling over exposure
• Configure WSO2 API Manager analytics
• Configure analytics
• Install additional cApp
• Implementing workflow extensions
• Deploy extended workflow
• Configure the billing engine
• Configure WSO2 API Manager
• Test the system
• Conclusion
• References

Prerequisites

Introduction

WSO2 API Manager provides a wide range of capabilities to manage, govern and monetize APIs. In this article we will provide a brief introduction to WSO2 API Manager monetization and how you can enable it for your billing system. WSO2 API Manager is a WSO2 carbon based server with Apache 2.0 license, which means it’s 100% open source and free of charge. WSO2 API Manager is composed of several sub-components and shipped with the following:

• API Publisher - implements API define rules and publishing
• API Store - API consumers subscribe and get access to the API
• Gateway- API governing and access control
• Key Manager - to secure APIs

WSO2 API Manager provides the following functionalities by default:

• API publishing
• API store (developer portal)
• Subscription management
• Token management
• Throttling
• Analytics
• Scalable deployment

There are also many extension points and standard extensibility that support most technologies and tools.

Monetization with WSO2 API Manager

API publisher implements and publishes the APIs to a gateway by specifying throttling tiers and related security and governance configuration. Allowed users can list the APIs available in the gateway using the API store. These subscribers can subscribe to these available APIs and invoke them.

In this billing model, API subscribers get billed based on the subscribed tiers. The throttling tiers act as a billing plan in WSO2 API Manager when invoked. In order to define a new billing plan, users need to define a new tier plan in WSO2 API Manager to reflect those business rules.

New throttling tiers contain the following properties

• Name - name of the tier
• Request count - allowable requests per unit time
• Unit time
• Stop on quota reached - enable or disable requests after quota reached

How subscriptions happen

Unlike the default subscription, in this model subscriptions should be managed by the billing system. In order to achieve this, you need to use the WSO2 API Manager subscription workflow. In the default workflow, subscriptions are automatically active. But when integrating with the billing engine, subscriptions are active if and only if the billing engine accepts the user. This behavior can be achieved by extending the default subscription workflow.

When a subscriber tries to subscribe to an API after the subscription workflow is configured, they will be redirected to the billing engine for authentication with the API manager workflow details. Then the billing engine accepts the user or signs them up as a new user to the system. After the user enrollment is done by the billing engine it will redirect them back to the WSO2 API manager and activate the subscription.

Data retrieving model for monetization

Since usage statistics are used for invoicing, it’s essential to collect data from WSO2 API Manager. You can easily do this by combining WSO2 API Manager with WSO2 Data Analytics Server (WSO2 DAS) analytics configurations. WSO2 DAS provides an enterprise-wide data analytics platform. WSO2 API Manager’s analytics is also powered by WSO2 DAS. All the analytics events are fired into WSO2 DAS and aggregated. Since data is collected at a single point, it’s easier to integrate analytics with billing engines.

WSO2 DAS and WSO2 API Manager need to be configured for analytics. There are two ways to do so. When configured for analytics all events related to usage in WSO2 API Manager will be collect by WSO2 DAS.

Following are the events that are sent to WSO2 DAS:

• Org.wso2.apimgt.statistics.request
• Org.wso2.apimgt.statistics.response
• Org.wso2.apimgt.statistics.fault
• Org.wso2.apimgt.statistics.throttle
• Org_wso2_apimgt_statistics_destination
• org.wso2.apimgt.statistics.workflow

For more information about the event details please refer here.

Data can be retrieved from WSO2 DAS either from the REST API or RDBMS client. To do so please refer here.

Controlling non-billing developers

A developer can subscribe to an API, invoke it and get invoiced for it but avoid the bill payment. To stop this from happening there needs to be a mechanism that addresses this issue. With WSO2 API Manager subscriptions can be enabled and disabled at the publisher’s will. If the user doesn’t pay, the publisher can disable the subscription and re-enable it once the payment is made. This can be achieved in two ways.

1. UI based enable/disable can be done in the publisher subscriptions section as shown below:
   

   Figure 1: Enable/disable subscription using UI

2. API based enable/disable can be done using the following URLs:
   • https://localhost:9443/aPI/am/publisher/v0.9/subscriptions/block-subscription
   • https://localhost:9443/aPI/am/publisher/v0.9/subscriptions/unblock-subscription

Controlling over exposure

Since the stop on quota reach option is available in throttling tier, the gateways can get flooded with requests. We can address this situation by using the API hard level throttle. With this option, the publisher can define the maximum requests per minute limit. This will ensure that the gateway doesn’t flood with too many requests even if the API publisher allows users to access APIs in an unlimited manner.

Configure WSO2 API Manager analytics

As mentioned earlier, the billing engine retrieves analytics data from WSO2 DAS. In order to do so, WSO2 API Manager should be configured for analytics with WSO2 DAS. Once that’s done, API invocation related events are published to WSO2 DAS, which then persists all the events and makes them available in internal tables. You may also want summarized data sets based on the raw event data to help the billing engine work efficiently. WSO2 DAS provides spark analytics so by using an analytics script you can summarize the raw event data into more meaningful data.

Configure analytics

To learn how to configure analytics using the REST client read this blog. To do the same using RDBMS client read this blog. In this article we have used the REST client since it is straightforward to configure.

Install additional cApp

You can use this additional cApp to generate additional summary data used to feed the billing engine. The WSO2 DAS REST API was used to retrieve the data.

• Locate cApp from apim-billing-engine/src/main/resources/DAS-cApp/dist/APIM_Billing_1.10.0.car
• Log in to WSO2 DAS web console
• Go to carbon-application section and deploy the downloaded cApp

Implementing workflow extensions

As explained earlier we need to customize and extend the default subscription workflow. To do this start a new maven project and create a new class name “SubscriptionBillingWorkflow” as shown below. The complete project can be found here.

/*
 *   Copyright (c) 2016, WSO2 Inc. (https://www.wso2.org) All Rights Reserved.
 *
 *   WSO2 Inc. licenses this file to you under the Apache License,
 *   Version 2.0 (the "License"); you may not use this file except
 *   in compliance with the License.
 *   You may obtain a copy of the License at
 *
 *     https://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing,
 *  software distributed under the License is distributed on an
 *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 *  KIND, either express or implied.  See the License for the
 *  specific language governing permissions and limitations
 *  under the License.
 *
 */
package org.wso2.sample.apimgt.workflow;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.wso2.carbon.apimgt.api.APIManagementException;
import org.wso2.carbon.apimgt.api.WorkflowResponse;
import org.wso2.carbon.apimgt.impl.APIConstants;
import org.wso2.carbon.apimgt.impl.APIManagerConfiguration;
import org.wso2.carbon.apimgt.impl.dao.ApiMgtDAO;
import org.wso2.carbon.apimgt.impl.dto.SubscriptionWorkflowDTO;
import org.wso2.carbon.apimgt.impl.dto.WorkflowDTO;
import org.wso2.carbon.apimgt.impl.workflow.*;
import org.wso2.carbon.utils.CarbonUtils;
import org.wso2.sample.apimgt.dao.BillingDao;

import java.io.File;
import java.util.List;

public class SubscriptionBillingWorkflow extends WorkflowExecutor {

    private static final Log log = LogFactory.getLog(SubscriptionBillingWorkflow.class);

    private String billingEngineUrl;
    private String APIMStoreUrl;

    @Override
    public String getWorkflowType() {
        return "SubscriptionBillingWorkflow";
    }

    @Override
    public List getWorkflowDetails(String status) throws WorkflowException {
        return null;
    }

    @Override
    public WorkflowResponse execute(WorkflowDTO workflowDTO) throws WorkflowException {
        super.execute(workflowDTO);

        SubscriptionWorkflowDTO subsWorkflowDTO = null;
        boolean userExists = false;

        if (workflowDTO instanceof SubscriptionWorkflowDTO) {
            subsWorkflowDTO = (SubscriptionWorkflowDTO) workflowDTO;

            try {
                BillingDao billingDao = BillingDao.getInstance();
                userExists = billingDao.userExists(subsWorkflowDTO.getSubscriber());
            } catch (APIManagementException e) {
                log.error("Error occurred while accessing Database: " + e.getMessage(), e);
                throw new WorkflowException("Error occurred while accessing Database: " + e.getMessage(), e);
            }
        }

        if (!userExists) {
            loadDefaultConfig();
            HttpWorkflowResponse httpworkflowResponse = new HttpWorkflowResponse();
            httpworkflowResponse.setRedirectUrl(billingEngineUrl);
            httpworkflowResponse.setAdditionalParameters("CallbackUrl",
                    APIMStoreUrl + "/site/blocks/workflow/workflow-listener/ajax/workflow-listener.jag");
            httpworkflowResponse.setAdditionalParameters("workflowRefId", workflowDTO.getExternalWorkflowReference());
            httpworkflowResponse.setAdditionalParameters("reDirectUrl", APIMStoreUrl);
            httpworkflowResponse.setRedirectConfirmationMsg(
                    "You will be redirected to a page to setup your billing " + "account Information");
            return httpworkflowResponse;
        } else {
            ApiMgtDAO apiMgtDAO = ApiMgtDAO.getInstance();
            try {
                apiMgtDAO.updateSubscriptionStatus(Integer.parseInt(workflowDTO.getWorkflowReference()),
                        APIConstants.SubscriptionStatus.UNBLOCKED);
            } catch (APIManagementException e) {
                log.error("Could not complete subscription creation workflow", e);
                throw new WorkflowException("Could not complete subscription creation workflow", e);
            }
        }

        return new GeneralWorkflowResponse();
    }

    @Override
    public WorkflowResponse complete(WorkflowDTO workflowDTO) throws WorkflowException {
        workflowDTO.setUpdatedTime(System.currentTimeMillis());
        super.complete(workflowDTO);
        log.info("Subscription Creation [Complete] Workflow Invoked. Workflow ID : " + workflowDTO
                .getExternalWorkflowReference() + "Workflow State : " + workflowDTO.getStatus());

        ApiMgtDAO apiMgtDAO = ApiMgtDAO.getInstance();

        if (WorkflowStatus.APPROVED.equals(workflowDTO.getStatus())) {
            try {
                apiMgtDAO.updateSubscriptionStatus(Integer.parseInt(workflowDTO.getWorkflowReference()),
                        APIConstants.SubscriptionStatus.UNBLOCKED);
            } catch (APIManagementException e) {
                log.error("Could not complete subscription creation workflow", e);
                throw new WorkflowException("Could not complete subscription creation workflow", e);
            }
        } else if (WorkflowStatus.REJECTED.equals(workflowDTO.getStatus())) {
            try {
                apiMgtDAO.updateSubscriptionStatus(Integer.parseInt(workflowDTO.getWorkflowReference()),
                        APIConstants.SubscriptionStatus.REJECTED);
            } catch (APIManagementException e) {
                log.error("Could not complete subscription creation workflow", e);
                throw new WorkflowException("Could not complete subscription creation workflow", e);
            }
        }
        return new GeneralWorkflowResponse();
    }

    public void loadDefaultConfig() {
        APIManagerConfiguration configuration = new APIManagerConfiguration();
        String filePath = CarbonUtils.getCarbonHome() + File.separator + "repository" +
                File.separator + "conf" + File.separator + "api-manager.xml";
        try {
            configuration.load(filePath);
        } catch (APIManagementException e) {
            log.error("cannot find the congifuration file at: " + filePath, e);
        }

        billingEngineUrl = configuration.getFirstProperty("billingEngineUrl");
        APIMStoreUrl = configuration.getFirstProperty("APIStore.URL");
    }
}

Deploy extended workflow

1. Build the maven project.
2. Copy and paste the built jar into the /repository/component/lib file.
3. Log into the WSO2 API Manager web console and browse the resources.
4. Edit the /_system/governance/apimgt/applicationdata/workflow-extensions.xml file.
5. Find the “SubscriptionCreation” tag and replace it with <SubscriptionCreation executor="org.wso2.sample.apimgt.workflow.SubscriptionBillingWorkflow"/>.

Configure the billing engine

1. Get the sample billing engine provided for WSO2 API Manager from here.
2. Build the project and locate the war file.
3. Deploy the war in the Tomcat container.
4. Locate the deploy webapp and edit the <apim-billing-engine-home>/WEB-INF/classes/datasource.properties file as shown below
   • url=jdbc:mysql://localhost:3306/billing
   • username=root
   • password=pass
   • driverClassName=com.mysql.jdbc.Driver
   • dialect=org.hibernate.dialect.MySQL5InnoDBDialect
   • apimStoreUrl=https://localhost:9443/
   • apimUserName=rukshan
   • apimPassword=rukshan@123
   • dasUrl=https://localhost:9444/
   • dasUserName=admin
   • dasPassword=admin
   • jksPath=<apim-billing-engine-home>/src/main/resources/wso2-jks/wso2carbon.jks
5. Define the billing plan
   • There are two different billing plans in the system to reflect the WSO2 API Manager billing model:
     • One is the standard usage plan, which charges based on the subscription fee and fee per additional request.
       

       Figure 2: Define standard billing plan

     • The other is the usage plan, which charges per request.
       

       Figure 3: Define usage billing plan

Configure WSO2 API Manager

1. Define the data source for the billing engine. Since the workflow extension is used we need to define the billing engine user details data source configuration in the /repository/conf/datasources/master-datasources.xml file as shown below:

   <datasource>
      <name>BILLING_DB</name>
      <description>The datasource used for registry and user manager</description>
      <jndiConfig>
          <name>jdbc/BILLING_DB</name>
      </jndiConfig>
      <definition type="RDBMS">
          <configuration>
              <url>jdbc:mysql://localhost:3306/billing?autoReconnect=true&amp;</url>
              <username>root</username>
              <password>pass</password>
              <driverClassName>com.mysql.jdbc.Driver</driverClassName>
              <maxActive>50</maxActive>
              <maxWait>60000</maxWait>
              <testOnBorrow>true</testOnBorrow>
              <validationQuery>SELECT 1</validationQuery>
              <validationInterval>30000</validationInterval>
              <defaultAutoCommit>false</defaultAutoCommit>
          </configuration>
      </definition>
   </datasource>
   

   • The extension also needs to know the billing engine web app URL. This should be added to the /repository/conf/api-manager.xml file as shown below:

     <billingEngineUrl>https://localhost:8080/apim-billing-engine-1.0-SNAPSHOT/app/main</billingEngineUrl>

   • Since WSO2 API Manager is configured for monetization we can enable the API status as free or premium. To enable this flag edit the registry location /_system/config/apimgt/applicationdata/tenant-conf.json as shown below:
     • To enable monetization set the “EnableMonetization” property to true.
     • To define if the unlimited tier is paid, set the IsUnlimitedTierPaid property to true.
   • Copy and paste the mysql jar to /repository/component/lib since the extension workflow uses the mysql connector.

Test the system

1. Start WSO2 API Manager, WSO2 DAS and Tomcat on respective ports
2. Log in to WSO2 API Manager admin-dashboard and define the new throttling tier as shown below:
   

   Figure 4: Define new tier

3. Log in to the publisher and create a new API or deploy the sample API.
   

   Figure 5: Deploy sample API

4. Log in to WSO2 API Manager store and try to subscribe to the API you just published.
   

   Figure 6: Subscribe to the API

5. WSO2 API Manager will ask your permission to redirect to the billing engine.
   

   Figure 7: Permission to redirect

6. Since you are not in the billing system signup as a new user.
   
   

   Figure 8: Sign up user

7. When you’re done, the billing engine will redirect you back to the WSO2 API Manager store with a successful subscription.
   

   Figure 9: Subscription state

8. Log in to the billing engine again with the user credential you just created.
   

   Figure 10: Sign in to billing engine

9. Define the new billing plan according to the tiers available in WSO2 API Manager.
   

   Figure 11: Define diamond billing plan

10. Invoke the API you just subscribed.
11. Go to the billing engine and generate an invoice as shown below (wait a few minutes to generate the summary data in WSO2 DAS).
    

    Figure 12: Start generating the invoice

    

    Figure 13: Generate invoice for the month

12. You can get the invoice based on the usage.
    

    Figure 14: Generated invoice based on usage

Conclusion

This article described the most basic and essential part of enabling monetization with WSO2 API Manager and WSO2 DAS. We used the sample billing engine here to demonstrate the billing engine functions. By using an advanced billing engine you can define more advanced billing rules and generate advanced invoices. Here we used WSO2 DAS REST API or RDBMS for retrieving data to the billing engine for API usage monitoring. By writing more specific queries you can generate custom data and more advanced invoices.

References

• [1] https://docs.wso2.com/display/AM1100/Configuring+HTTP+Redirection+for+Workflows
• [2] https://docs.wso2.com/display/AM1100/Configuring+API+Monetization+Category+Labels