[Article] WSO2 API Manager Deployment Customization and Troubleshooting - Part 2

  • Sanjeewa Malalgoda
  • Director - Engineering | Architect at WSO2 - WSO2
  • 18 Mar, 2015


WSO2 API Manager is a complete solution for publishing APIs, creating and managing a developer community, and for scalably routing API traffic. It leverages proven, production-ready, integration, security, and governance components from WSO2 Enterprise Service Bus, WSO2 Identity Server, and WSO2 Governance Registry. Moreover, it is powered by WSO2 Business Activity Monitor, thereby making WSO2 API Manager ready for any large-scale deployments right away.

When you work on WSO2 API Manager deployments you need to follow deployment best practices. Generally, with API management platform deployment, we need to focus on some customization that's specific to your use case. Moreover, you may also need to fine-tune some parameters according to specific requirements and system load. When you configure deployment, you may come across certain issues, and when deployment is up and running you may get some run-time issues due to various reasons. If you have some knowledge about the product and carbon platform, you might be able to sort out these issues fairly easily.


Applies to

WSO2 API Manager Version 1.8.0 and above

Table of contents

  1. Stats and usage data-related issues
  2. Distributed deployment-specific issues
  3. API invocation-related issues
  4. Other common issues
  5. Conclusion


Stats and usage data-related issues

When you work with usage data-related issues, first you should check the data source configurations in WSO2 BAM and WSO2 API Manager. This is important because API publisher will directly use this data source configuration to connect the stats database.

We use the stats database to store all summarized usage data. This database should be shared between the WSO2 API Manager node and the BAM node as both servers need to access tables. The WSO2 API Manager usage client will call the stats database directly and pull usage information. Later, this usage data will be passed to the presentation layer.

Then we need to check the created tables in the usage database. Most times, we've found problems here that stem from configuration issues. Moreover, we need to focus on the same configurations when we troubleshoot issues related to billing sample.

API manager usage stats database table creation

-Dsetup does not work for usage tables. The usage table definitions are not available in setup scripts.
In WSO2 API Manager and WSO2 BAM deployment, we need a user manager database, registry database, and API manager database. We have setup scripts for those databases under the db scripts folder in product distribution. There is no need to create any tables in the stats database (manually or using setup script) as the API manager toolbox (deployed in BAM) will create them when hive queries are executed. -Dsetup option will not apply to the hive scripts inside the toolbox deployed in BAM. If you have deployed the toolbox properly and configured data sources, the table will be created after the first hive script runs.


Distributed deployment-specific issues

Connectivity between components in distributed deployment

We need to understand connectivity between components in distributed API manager deployment clearly.
This is important when you troubleshoot issues related to distributed API Manager deployment. Refer to the following steps to understand connectivity between components.

  • Changed the admin password
  • Tried to log in to publisher and got an insufficient privilege error.
  • Then changed the admin password in authManager element in api-manager.xml
  • Restarted and was able to login to API publisher.
  • In the API manager store and publisher, we need to specify auth manager details properly. In distributed deployment, we normally point to the key manager as the auth server. When the user tries to login, the system publisher/store will connect to the auth server to authenticate the user.
  • Then, we created an API and tried to publish. A permission error occurs again.
  • Then, we changed the password under the API gateway element in api-manager.xml
  • Restarted and published the API.
  • In the API publisher node, we need to configure the gateway URL and credentials properly. This information will be used to call admin services deployed in the gateway. When we create an API from the publisher and try to publish it to the gateway, we do a web service call to push API configuration in the gateway node.
  • Then, we tried to invoke an API using an existing key.
  • Got the key validation error
  • We then changed the admin password in KeyManager element in api-manager.xml and all issues got resolved.
  • Each and every request that comes to the gateway needs to pass the API authentication handler. In this handler, we will extract the required information and call the key validation service in the key manager server. For this, we need to configure the key manager server URL and credentials properly in api-manager.xml file in the gateway node.

Why thrift key validation is not recommend in production deployments

Thrift key validation does not work when we have a load balancer fronted key manager.
The reason for this is that most load balancers are not capable of routing traffic in a session-aware manner.
Therefore, in such cases, it's always recommended to use the WS key validation client. When we use web service clients we need to enable a sticky session at the load balancer level. Moreover, we need to give a session cookie that we've used to handle session stickiness.

Accessing APIs by single user using multiple devices/clients

As you may know, we allow one user to have one access token (for a given application) at a given time. But when we use multiple devices, the token may be revoked by one application. Therefore, users may request to have multiple active access tokens at a given time. However, we designed WSO2 API Manager to have one active token for application user combination. Hence, these types of requirements should be addressed by modifying the client (end-user application) according to custom requirements. This article [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-configure-wso2-api-manager-to.html] explains how you can achieve this.

What if the server goes OOM or starts spinning

In the event a server goes OOM, you would need to get a heap dump. It is normally dumped by default, if not, we have to dump manually. To find CPU spinning issues, get the thread dump and thread usage and try to find the culprit thread. Refer to this article [https://nirmalfdo.blogspot.com/2013/05/how-to-find-culprit-when-cpu-starts-to.html] for instructions.

Why APIs are not visible in the store in distributed deployment

Sometimes users may complain that they cannot see APIs in the store UI once they publish it from publisher. The API store and publisher will share the same registry. If the registry cache is not synchronized properly, then you will see some inconsistencies. Refer to the following instructions if you encounter such an issue.

  • Change the admin password
  • Store, Publisher should be in the same clustering domain
  • Cache ID should be correctly specified in registry.xml
  • If configuration is done correctly then, users can log into the management console and see if the APIs are indexed correctly
  • Registry indexing changes can cause to this issue

Why some changes made to APIs are not immediately reflected in the gateway

On the API gateway side, the API resource cache will have API metadata. If that's enabled, the changes on Auth type, and the addition of resources and changes in resource level throttling policies will not take effect until a cache expiration occurs. The default cache expiration is 15 minutes and you can try the same after that time limit.

Move API manager deployment between environments

Another common issue found in deployments is how to move one API manager deployment to another.
As of now, we don't have a one-step migration process. However, it can be done using the following steps listed here [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-move-wso2-api-management.html]
We have planned to introduce deployable API artifacts for WSO2 API Manager in the future.


API invocation-related issues

Fixing no matching resource found issues

Once the request comes to the gateway, first we will find a matching API by looking at the URL of the incoming request. Once the API request is dispatched to the correct API, it should be mapped to the resources available in the API. For this, we should understand how resource mapping works. These [https://sanjeewamalalgoda.blogspot.com/2014/03/fixing-issue-in-wso2-api-manager-due-to.html], [https://charithaka.blogspot.com/2014/03/common-mistakes-to-avoid-in-wso2-api.html] articles may help you to fix some common issues.

URL mapping and URI template

If you create an API from the API publisher application, then it will create the URL mapping for API resources (up to WSO2 API Manager 1.7.0). From WSO2 API Manager 1.7.0 onward, we will create the URL mapping or URI template based on user input. Moreover, note that you can use uri-template for this according to your requirements. Uri-template is more powerful and allows you to define complex resource mappings.
If you define the URL Template as /sanjeewa*
Then it will be mapped to the following requests.

  • /sanjeewa/admin?test=done
  • /sanjeewa/test
  • /sanjeewa/
  • /sanjeewa

Refer to these [https://svn.wso2.org/repos/wso2/carbon/platform/branches/turing/dependencies/commons/uri-templates/1.6.5/src/test/java/org/wso2/uri/template/parser/ParserTest.java] tests to understand how complex mappings work.

Call multiple APIs from a single API request

Another common requirement is calling multiple APIs from a single API call.
Most users are interested in this because it minimizes network latency for multiple API calls. For this, we can use the batch API/API chaining approach. For further information, refer to this [https://sanjeewamalalgoda.blogspot.com/2014/03/batch-api-api-chaining-scenario.html] article.


Other common issues

When we have public deployment how we can let users tryout APIs with the global key concept?

When users opt for a public API store, they need to let their users try the API without subscribing to it (i.e. without getting an access token). If you ask them to login, create an application, subscribe, obtain a token, and use it, they would need to spend a significant amount of time to invoke the API. Therefore, it would be better to allow them to invoke APIs with one click, even without login. However, you need to apply the correct throttling to avoid misuse of APIs.
This capability is available in most of publicly available API stores. This [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-customize-wso2-api-manager.html] article describes how you can customize our API manager to address that requirement.

Deleting APIs

We noticed that some users try to remove APIs by deleting registry API artifact + synapse config and database entry. This is a wrong move. We should always ask users to delete APIs using the publisher user interface. If you delete the API part by part (from registry, database, and synapse) then you will get multiple issues due to active subscriptions, resource mapping, and other dependencies.
Sometimes when you delete registry artifacts, API artifacts may get deleted due to user errors. If we delete API artifact from the registry, it will cause issues in the API store and publisher. When we load APIs in store/publisher applications, we normally retrieve resources from registry. If API data in the registry gets deleted due to some reason, we can fix it by following the instructions listed here [https://sanjeewamalalgoda.blogspot.com/2013/11/how-to-fix-issue-in-subscription-page.html].

Multiple level throttling in API manager

Another common issue is tuning parameters and minimum deployment requirements. We can use the instructions listed in these articles [https://docs.wso2.org/display/AM150/Performance+Tuning+Recommendations][https://sanjeewamalalgoda.blogspot.com/2013/11/recommended-system-requirement-for-wso2.html] when we setup deployment. Moreover, we can apply ESB tuning parameters to the API gateway as it's almost similar to ESB.

Minimum hardware requirements and tuning parameters

The API manager has throttling in 3 levels. We introduced resource level throttling in WSO2 API Manager 1.5.0; here [https://sanjeewamalalgoda.blogspot.com/2013/09/throttling-support-at-api-resource.html] is a brief description on how it works.

Writing test cases

From WSO2 API Manager 1.7.0 onwards, you will be able to write Java test cases easily as we expose common API management operations as Java utility methods (you can find the store and publisher rest client source code here [[12]https://svn.wso2.org/repos/wso2/carbon/platform/branches/turing/products/apimgt/1.7.0/modules/integration/tests/src/test/java/org/wso2/carbon/am/tests/util/]). We will also be able to add host object functionalities for automated tests (refer to this sample tests [https://svn.wso2.org/repos/wso2/carbon/platform/branches/turing/products/apimgt/1.7.0/modules/integration/tests/src/test/java/org/wso2/carbon/am/tests/sample/HostObjectTestCase.java]). These kinds of integration tests are important because you don't have to test all functionality manually once you make some change to the system.



In this article series, we discussed the basics of WSO2 API Manager deployments. This part of the two-part article mainly focused on deployment troubleshooting. We discussed deployment best practices and fine tuning parameters as well. When you deploy production deployment, the minimum down time is a key element that you would need to pay attention to. With this, you should be able to sort out any issues that crop up within a short period of time.



[2] https://docs.wso2.com/display/AM180/WSO2+API+Manager+Documentation


About Author

  • Sanjeewa Malalgoda
  • Director - Engineering | Architect at WSO2
  • WSO2