Message Transformation with WSO2 API Microgateway
- Dushani Wellappili
- Technical Lead - WSO2
WSO2 API Microgateway is a cloud-native lightweight message processor used to expose microservices as APIs. An API hosted in API Microgateway receives requests sent from the consumer application and routes it to the backend service. It then receives the response sent from the backend and routes it back to the consumer application.
In the real world, message consumers and producers use different techniques according to their requirements. So, your consumer application and backend do not always agree on a common message format. This is where message transformation comes into play. WSO2 API Microgateway is designed to allow you to extend the default mediation flow. If you need to modify or transform the API requests and responses, you can write your own request/response interceptor as a Ballerina function, intercepting the default flow. Microgateway allows you to add request/response interceptors in the following levels:
- Globally for the API
- Per API resource
When API requests come into WSO2 API Microgateway, it first goes through the global API level request interceptor, then to the resource level request interceptor, and finally to the backend. Once the response is received by the Microgateway from the backend service, it first goes through the resource level response interceptor, then the global API level response interceptor, and finally back to the consumer application.
A few common scenarios where message transformation is needed in the Microgateway layer are as follows:
- The client application and backend service use different types of message content
- Passing additional information/modifying existing information in the request/response payload, which the client/backend is unable to send
- Adding additional headers/modify existing headers passed in the request/response
- Tracing/debugging the message flow in Microgateway by logging the message payload, headers, timestamp, etc.
- Validating the request and response payloads
- Based on the request message header or payload content, the request needs to be routed to a different backend
- Invoke other endpoints and obtain the data required to build the request that needs to be sent to the actual API backend
Example 1:
Let’s assume that your backend API expects the request payload to be in XML format, but your client application can only send messages in JSON format. In such cases, message transformations in Microgateway can help you transform the request payload from JSON to XML format using a request interceptor. For example, in the event that you have an employee record management backend service that expects the “Add employee” resource payload to be in XML format, this backend service is exposed as an API via API Microgateway. However, the client application is only capable of sending JSON payloads.
Here, we can use a request interceptor in Microgateway to convert the format of the incoming request payload from JSON to XML according to the backend service. This request interceptor is written as a Ballerina function to perform the type conversion in the payload of http:Request Ballerina object. Refer [1] to learn more about the Ballerina http module and its capabilities.
import ballerina/io; import ballerina/http; public function transformJsonToXml (http:Caller outboundEp, http:Request req) { io:println("Request is intercepted."); io:println("Initial content type : ", req.getContentType()); var jsonPayload = req.getJsonPayload(); if (jsonPayload is json) { var xmlPayload = jsonPayload.toXML({}); if (xmlPayload is xml) { req.setPayload(untaint xmlPayload); } } io:println("After transformation content type : ", req.getContentType()); io:println("After transformation request payload ", req.getXmlPayload()); }
The above interceptor needs to be saved as a Ballerina file with the “.bal” extension inside the interceptor’s directory of the relevant Microgateway project. You can provide a file name of your choice. Assuming that we need to intercept only the requests coming for “POST /employee” resource, the interceptor is attached to the particular resource level in the Open API definition. (Refer below for a sample OpenAPI definition “.yaml” file inside api_definitions directory of the Microgateway project.)
... paths: /employee: post: tags: - employee summary: Add employee record operationId: addEmployee x-wso2-request-interceptor: transformJsonToXml responses: "200": description: successful operation ... ...
The resource level request interceptor is engaged using the Open API extension field “x-wso2-request-interceptor”.
During the invocation, we can check the Ballerina HTTP trace logs to verify if the request going to the backend contains an XML payload instead of JSON. To enable HTTP trace logs, run the microgateway as follows:
bash gateway -e b7a.http.tracelog.console=true /home/dushaniw/WSO2/Microgateway/employee/target/employee.balx
Once the API is invoked, the following trace log in the console shows the incoming request payload in JSON.
As seen here, after going through the interceptor, the request payload has been converted to XML according to the following trace log:
Example 2:
Let’s assume the WeatherAPI backend can only send XML responses for all API operations. However, the client application always expects the response in JSON format. (You have no control over a third-party WeatherAPI backend service to change the behavior.)
In this event, you need to convert the XML response received from the backend to JSON during the response flow in Microgateway. Engaging this response interceptor globally to WeatherAPI will make sure that the XML-to-JSON transformation will be applied to all API responses received for this WeatherAPI deployed in Microgateway.
import ballerina/io; import ballerina/http; public function transformXmlToJson (http:Caller outboundEp, http:Response res) { io:println("Response is intercepted."); io:println("Initial content type : ", res.getContentType()); var xmlPayload = res.getXmlPayload(); if (xmlPayload is xml) { var jsonPayload = xmlPayload.toJSON({attributePrefix: "#", preserveNamespaces: false}); res.setPayload(untaint jsonPayload); } io:println("After transformation content type : ", res.getContentType()); io:println("After transformation request payload ", res.getJsonPayload()); }
openapi: 3.0.0 info: description: This is a mock backend for weather service version: v1 title: Weather Service x-wso2-basePath: /weatherservice/v1 x-wso2-production-endpoints: urls: - "https://samples.openweathermap.org/data/2.5" x-wso2-response-interceptor: transformXmlToJson paths: /weather: get: parameters: - name: q in: query ...
According to the following trace log, the response coming into the Microgateway is in XML format:
However, after going through the response transformation interceptor, the downstream trace log shows the response going back to the client in application/json format.
Example 3:
If there are multiple types of client applications accessing your API published in the Microgateway and each application uses different messaging formats, you need to transform the request and response payloads accordingly when passing through the Microgateway. For example, consider a client who is accessing XYZ bank’s APIs and checking their account balance online. The API developer wants to allow the client’s mobile device to access a select set of account details and expose all data to other desktop application clients.
This can be achieved by having both request and response interceptors in Microgateway to read the “User-Agent” transport header inside the request interceptor and modify the response payload accordingly.
Refer below for a request interceptor that extracts the User-Agent header value from the http:Request and stores it in the attribute map of the InvocationContext dataholder. An IncovationContext data holder is created per request and preserved for a single request-response flow [2].
import ballerina/io; import ballerina/http; import ballerina/runtime; public function interceptRequest (http:Caller outboundEp, http:Request req) { io:println("Request is intercepted to read User-Agent header."); runtime:getInvocationContext().attributes["CLIENT_ACCESS_DEVICE"]= untaint req.userAgent; }
Next, the following response interceptor will modify the receiving JSON payload according to the previously read User-Agent value during the request path.
import ballerina/io; import ballerina/http; import ballerina/runtime; public function transformResponse (http:Caller outboundEp, http:Response res) { io:println("Response is intercepted to remove address field for iphone users."); any device = runtime:getInvocationContext().attributes["CLIENT_ACCESS_DEVICE"]; if (res.statusCode == 200) { if (device is string) { if (device == "iPhone") { var response = res.getJsonPayload(); if (response is json) { response.remove("address"); res.setPayload(untaint response); } } } } }
After adding the above interceptors inside the interceptor’s directory in the Microgateway project, attach them to the Open API definition as follows:
openapi: 3.0.0 info: description: This is a mock backend for XYZ Bank version: 1.0.0 title: XYZBANK x-wso2-basePath: /xyzbank/v1 x-wso2-production-endpoints: urls: - "https://www.mocky.io/v2/5d2dc2b82e00005900c580f2" paths: /account/{accountId}: get: parameters: - name: accountId in: path description: accountId required: true schema: type: string summary: Get Account details by Id operationId: getAccount x-wso2-response-interceptor: transformResponse x-wso2-request-interceptor: interceptRequest responses: "200": description: successful operation content: application/json: schema: items: $ref: "#/components/schemas/Account" ...
Example 4:
In Microgateway, the keyword “Authorization” is a reserved name used for the header to pass the JWT/OAuth Token for the API. There may be a requirement for you to directly pass a custom key in the header “Authorization” to the backend. In such cases, there is an alternative to send the custom key under a separate header name. Then, using request transformations, extract the header value and set it to an “Authorization” header.
The request interceptor for the above scenario will be as follows:
import ballerina/io; import ballerina/http; public function setAuthHeaderInRequest (http:Caller outboundEp, http:Request req) { io:println("Request is intercepted."); If (req.hasHeader("X-Authorization")) { string customHeader = req.getHeader("X-Authorization"); req.setHeader("Authorization",customHeader); req.removeHeader("X-Authorization"); } }
The following trace log for an incoming request shows that both “Authorization” and “X-Authorization” headers are present in the request:
Yet, only the “Authorization” header (with the value of the “X-Authorization” header, which was in the incoming request) can be seen in the outgoing request from Microgateway.
Example 5:
When the client application sends API requests, it may be required to log certain attributes of the request or to assign a UUID as a header and log it during the message flow in order to trace the request and response in the Microgateway.
The following request interceptor is expected to generate a UUID and assign it as the CorrelationID header of the request, and log it in the same request flow:
import ballerina/http; import ballerina/time; import ballerina/log; import ballerina/runtime; public function logRequest (http:Caller outboundEp, http:Request req) { time:Time time = time:currentTime(); int correlationID = time.time; runtime:getInvocationContext().attributes["CORRELATION_ID"] = correlationID; log:printInfo("Request correlationID: " + correlationID); req.setHeader("CorrelationID",correlationID); }
The following response interceptor will engage in the response flow, log the corresponding CorrelationID, and send it as a response header to the client application:
import ballerina/http; import ballerina/runtime; public function logResponse (http:Caller outboundEp, http:Response res) { any correlateId = runtime:getInvocationContext().attributes["CORRELATION_ID"]; if (correlateId is int) { log:printInfo("Response correlationID: " + correlateId); res.setHeader("CorrelationID",correlateId); } }
The above interceptors are needed to engage in the API definition as follows:
openapi: 3.0.0 info: description: This is a mock backend for XYZ Bank version: 1.0.0 title: XYZBANK x-wso2-basePath: /xyzbank/v1 x-wso2-production-endpoints: urls: - "https://www.mocky.io/v2/5d2dc2b82e00005900c580f2" x-wso2-request-interceptor: logRequest x-wso2-response-interceptor: logResponse paths: /account/{accountId}: get: parameters: - name: accountId ...
The corresponding info log for the request interceptor and the trace log with the CorrelationID header passed to the backend are as follows:
The corresponding info log with the correlation ID in the response flow and the trace logs with CorrelationID header passed back to the client are as follows:
In addition to the above-discussed scenarios, there may be several other instances when you would need to intercept the request and response flows of the Microgateway. You can use interceptors to validate your request and response payloads before going to the backend or client respectively. You may have a requirement to route the client’s API request coming into the Microgateway to a specific endpoint based on some predefined conditions. There may also be cases where you need to invoke multiple endpoints to get the data to create an actual request to the API backend. For example, when a client uses a geolocation mobile application to search for nearby healthcare centers, it should provide information of nearby healthcare centers using the mobile’s current longitude and latitude. There are dedicated backends to provide the zip code once you provide the longitude and latitude, which in turn provides details of healthcare centers in the area. This type of a service requirement can also be handled by Microgateway using request interceptors.
In summary, API developers no longer need to worry about using different types of backends and consumer applications as WSO2’s Microgateway provides premium support to plug in external Ballerina functions as interceptors during its request and response outflows. Therefore, API developers can transform/modify or validate the API request/response payload, headers, etc. according to their requirement before sending it to the backend or responding to the client.