[Article] WSO2 API Manager Deployment Customization and Troubleshooting - Part 1

Introduction

When you work on API manager deployments, you need to follow deployment best practices. Generally with API management platform deployments, some amount of customization needs to be incorporated to address specific use cases. Moreover, some of the parameters too would need to be tweaked to meet certain requirements and system load. Once you configure deployment, most of then than not you may encounter certain issues. Thereafter, when the deployment is finally up and running too there could be come run-time issues due to various reasons. In these instances, it's useful to have some knowledge about the product and the carbon platform, which would help to address some of these issues fairly quickly. In this part of this article we will discuss in detail some of the most common issues that we've encountered during deployment and configuration processes of WSO2 API Manager.

WSO2 API Manager is a complete solution for publishing APIs, creating and managing a developer community, and for scalably routing API traffic. It leverages proven, production-ready integration, security, and governance components from WSO2 Enterprise Service Bus, WSO2 Identity Server, and WSO2 Governance Registry. Moreover, it is powered by WSO2 Business Activity Monitor, thereby making WSO2 API Manager ready for any large-scale deployment right away.

Applies to

Table of contents

1. Customization using extension points
2. WSO2 API Manager REST APIs and admin services
3. WSO2 API Manager with proxy and load balancers
4. Deployment-specific issues/Deployment patterns
5. Conclusion

Customization Using extension points

In WSO2 API Manager, we have provided many extension points to extend functionalities based on custom requirements. Therefore, when you configure deployment, you can add extensions to carry out deployment-specific tasks. Here, we've explained the common extension points available in WSO2 API Manager. For further more information, refer to the WSO2 API Manager documentation section.

• Workflow - Workflow extensions allow you to attach a custom workflow for various operations in WSO2 API Manager, such as user signup, application creation, registration, subscription, etc. If you want to have human interaction, approval process, or trigger some external system, you can use workflow extension points available in WSO2 API Manager (https://docs.wso2.com/display/AM180/Adding+Workflow+Extensions)
• JWT generation - In the WSO2 API Manager (default product JWT generation) we retrieve basic application-specific details, subscription details, and user information available on the system. However, when we have more complex use cases, we might need to send more information with JWT. For this, we have an extension point to implement a custom JWT generator according to the specific requirement
  (https://wso2.com/library/articles/2014/12/customize-json-web-token-generation-with-wso2-api-manager-1.8.0/)
• Claim retrieve - In WSO2 API Manager, you have the capability of plugin external claim retriever class. With this, you can generate claims and add them to JWT based on your requirement
• Adding new handlers - In WSO2 API Manager, you have 5 default handlers to do authentication, stats publishing, logging, etc. However, you may add new handlers according to your requirements. In this (https://docs.wso2.com/display/AM180/Writing+Custom+Handlers) article, we have discussed writing new handler for WSO2 API Manager
• Extension handlers - If you want to log all incoming and outgoing message parameters, then you need enable debug logs for WSO2 API Manager extension handler. Here we can capture all API management specific parameters and log them
• Mediation extension - The API Gateway has a default mediation flow that's executed in each API invocation. You can perform additional custom mediation for the messages in the API Gateway by extending its mediation flow. An extension is provided as a Synapse mediation sequence. In this (https://docs.wso2.com/display/AM160/Adding+a+Mediation+Extension) article, you can find more information about adding mediation extension
• How to pass 2 auth headers in the same request (one for WSO2 API Manager and one for backend). Sometimes we need to pass authentication headers to the backend server in addition to the bearer token used in the gateway. In such cases, we need to use the same authentication header for both. Hence, we can add some custom sequence and swap headers in the gateway after authentication. In this article (https://sanjeewamalalgoda.blogspot.com/2014/11/how-to-use-custom-authentication-header.html)
• In WSO2 API Manager distributed deployments, the gateway will talk to the key manager to validate incoming access tokens. Normally we use WS client or thrift client to do this. If you want to enable mutual SSL in-between, enable mutual SSL for server-to-server communication. We can use this for the API Manager gateway to communicate with the key manager and validate the call (https://sanjeewamalalgoda.blogspot.com/2014/11/how-to-enable-mutual-ssl-connection.html)
• Throttling - WSO2 API Manager supports throttling for APIs. Some production deployments need more complex throttling mechanisms like IP-based throttling and multiple layer throttling. In this (https://sanjeewamalalgoda.blogspot.com/2014/12/how-to-use-two-layer-throttling-in-wso2.html) article, we have discussed multiple layer throttling configuration and in this (https://docs.wso2.com/display/AM170/Managing+Throttling+Tiers) article, we discussed IP based throttling. WSO2 API Manager uses WSO2 throttle complement; you may find further information in this (https://wso2.com/library/articles/wso2-throttling/) article

WSO2 API Manager REST APIs and admin services

If you have customized application and need to call the API management server, then our rest API and admin service APIs might help you. Sometimes you may need to create API automatically when you create a new service in some application server. To do this, you may automate the API creation process for service creation. For this type of requirement you can call our API publisher rest APIs and create APIs. Almost all operations exposed in the API manager server can be done via web service calls. Some of these operations are exposed as rest APIs and some of them are exposed as admin services. Rest APIs and admin services are exposed to external users to perform API manager functionalities. Refer to the links below to view documentation pages for available APIs in WSO2 API Manager.

https://docs.wso2.com/display/AM180/Publisher+APIs

https://docs.wso2.com/display/AM180/Store+APIs

https://docs.wso2.com/display/AM180/Token+API

https://docs.wso2.com/display/AM180/WSO2+Admin+Services

WSO2 API Manager with proxy and load balancers

Today most production servers are fronted with some kind of proxy service or load balancer. Therefore, many users front their API management deployment with a proxy or load balancer. As a result, we need to know how to configure an API manager with a reverse proxy. From WSO2 API Manager 1.8.0 onwards, we require a minimum set of configurations to configure the API manager with the proxy server. The following documents may help you to understand how it works and how to configure it.

https://docs.wso2.com/display/AM180/Adding+a+Reverse+Proxy+Server

How to run API manager with nginx reverse proxy

https://sanjeewamalalgoda.blogspot.com/2014/12/configure-wso2-api-manager-with-reverse.html

https://shavanthaw.blogspot.com/2014/06/configuring-nginx-for-wso2-carbon.html

With Apache2

https://vvratha.blogspot.com/2014/08/configuring-wso2-apimanager-with-apache.html

Deployment specific issues/deployment patterns

When you work with WSO2 API Manager distributed deployment you need to understand components, relationship between them, and data flow. This section we will briefly discuss these.
In distributed deployment, we deploy four components (gateway, key manager, store, and publisher) in different four servers. This is one of the most common deployment patterns.
In this section, we will discuss some of the common issues in distributed deployment and look at how we can solve them.
The following diagram explains connectivity between components.

Login failure

The following diagram explains connectivity between components for login call. As you can see, when the user tries to login to the system the call will first go to the auth server. Then the auth server will call LDAP and will then check the required permissions with the permission database. When you troubleshoot any issue, you would need to check the logs of each component. If you see an error in store/publisher node that indicates you cannot connect to the auth server, you will need to check the connectivity between store/publisher and auth server. Sometimes you may see errors when connecting to LDAP or permission database. In such cases, you need to check logs and fix whatever has been missed.

API invocation failures

In WSO2 API Manager, API invocation failures are other common types of issues. API invocation needs the following operations:

• Client request access token via API gateway
• In the API gateway, we have deployed a token API that's fronted OAuth2 app deployed in the key management server.
• Once the token generation request is received to auth app, it will validate the user and application details (consumer key and secret key)
• If everything is correct, return access token to user
• Then the user will have valid access token and may call the gateway
• API gateway accepts the API call and will go through handlers (first one would be auth handler)
• API gateway will talk to the key manager and validate the token
• If the user has a valid token, then the auth handler will pass the request and it will go to the throttle handler. Here we will check throttling counters and the current request and decide if the request can move forward or need to be blocked. If the request is not blocked, it will go through other handlers
• Once all handlers are passed, we can let the request go to the back-end server

So if you are troubleshooting any issue you need to be aware of these steps. Let's say some user complains about API invocation; firstly we would need to check the gateway logs. Then you can check if the request came to the gateway or not. If it came to the gateway, then check the key validation process. If the key validation fails due to a connection issue between the gateway and the key manager, then you need to fix it. If the validation call goes to the key manager, then there should be some log in the key manager. Likewise, you need to check each and every step when you troubleshoot issues.

Troubleshooting monitoring and workflow-related issues
The following diagram will show you how the components are connected in a stats-enabled deployment.

As illustrated in the diagram, when an API request comes in, the API gateway will collect the required information and send events to the business activity monitor (BAM). Then BAM will process them and write to the relational database. If you don't see stats first you need to check the gateway logs to check if there are any issues with the stats publishing part. If the stats are published to BAM, we need to check data in Cassandra. For this, you may use Cassandra explorer available in BAM. Thereafter you will publish the events. In the next step, you would need to check summary generation scripts running property. If these scripts are executed properly, it will create a stats-related table in the database pointed by the API manager stats data source. Now we know that the data is available in RDBMS, so we'll need to check the API manager publisher application. If we configure the stats database URL and credentials properly, then the publisher application can pull the required data from the database and visualize them. This is the way you would need to address deployment issues.

Conclusion

In this article, we discussed the basics of WSO2 API Manager deployment, customization, and troubleshooting. We mainly focused on the API manager extension points that we can use to implement customizations that are specific to deployment. Moreover, we discussed the different configuration options and some deployment-specific issues as well. The second part of this series will focus on deployment troubleshooting and discuss in detail deployment best practices and fine tuning parameters.

References

[2] https://docs.wso2.com/display/AM180/WSO2+API+Manager+Documentation