WSO2Con 2013 CFP Banner

[Article] RESTful Integration with WSO2 ESB

Discuss this article on Stack Overflow
By Kasun Indrasiri
  • 21 Jan, 2014
  • Level: Advanced
  • Reads: 18271

With the increasing usage of RESTful APIs, any integration platform needs to cater to the real-world RESTful integration needs. This articles gives you a comprehensive understanding of how WSO2 ESB can be used to realize all RESTful integration needs. In particular, this article targets how resources and associated operations (HTTP verbs) can be integrated with HTTP endpoints and the usage of payload factory with JSON support for JSON-JSON and XML-JSON transformation scenarios.

Kasun Indrasiri
Architect
WSO2 Inc

Applies to

WSO2 ESB 4.8.0
WSO2 AS 5.2.1

Table of contents

Introduction

Representational state transfer (REST) is becoming increasingly popular over the Web as a simpler alternative to SOAP web services. It was initially introduced by Roy Fielding as "Architectural Styles and the Design of Network-based Software Architectures"[1] which analyzes a set of software architecture principles that use the Web as a platform for distributed computing. Since then, it has been heavily adopted into most web-based software solutions and it pretty much supersedes SOAP in most web services-based solutions.

Source: Is REST successful in the enterprise? [2]

REST is an architecture style for designing networked applications or distributed hypermedia systems based on the Resource Oriented Architecture (ROA) paradigm.

REST ignores the details of component implementation and protocol syntax in order to focus on the roles of components, the constraints upon their interaction with other components, and their interpretation of significant data elements.

REST triangle

REST Triangle [3]

Nouns - Identification and addressability of resources

  • A 'resource' is a key abstraction REST and any information that can be named can be a resource.
  • Everything that service provides is a resource. For instance, a text file, document, image, a collection of other resources, a non-virtual object can be considered as resource. So, such interesting bits of information are identified with URIs and are usually accessed via a URL
  • Everything of interest should be named.
  • Separating nouns from verbs and representations.
  • Resources are defined by URIs:
  • http://example.com/customers/1234

    http://example.com/orders/2007/10/776654

    http://example.com/products/4554

    http://example.com/processes/salary-increase-234

Verbs - Operations that can be applied on a resources/CRUD

Verbs are generally used as operations, generally encompasses everything that can be done to a resource (or a piece of data). Mostly CRUD (Create, Read, Update, Delete) operations. In the HTTP scope, these are mapped to POST, GET, PUT and DELETE along with some additional verbs such as OPTIONS and HEAD.

  • GET: The HTTP GET method is used to retrieve a representation of a resource.
  • POST: The POST verb is most-often utilized for creation of new resources.
  • PUT: PUT is most-often utilized for update capabilities, PUT-ing to a known resource URI with the request request body containing the newly-updated representation of the original resource.
  • DELETE: Delete a resource identified by a URI.
  • HEAD: Similar to GET, but a response for a HEAD request must not have a body, though the meta information are identical to a response to a GET request for the same resource URI. This method is often used for testing hypertext links for validity, accessibility, and recent modification.
  • OPTIONS: This represents a request for information about the communication options available on the request/response chain identified by the Request-URI

Data formats - Identification and addressability of resources

Data formats provide the format for the data that will take part in your RESTful discussion. Most commons data types are XML JSON, or CSV. A resource, or resource state, is the data that defines the resource representation. Therefore, requests such as POST or PUT should contain the resource representation as the content body (or entity body). When it comes to returning representations, most REST APIs allow on-demand data representation, in which users can request the required returning representation. They could possibly use HTTP ‘Accept’ header to specify the required returning representation. However, most modern REST APIs use format specifiers, which looks more like a file extension. For instance GET http://www.example.com/customers.xml will return an XML representation while GET http://www.example.com/customers.json gives a JSON response representation.

In a nutshell, the REST architectural style, data and functionality are considered resources, and these resources are accessed using uniform resource identifiers (URIs), typically links on the web. The resources are acted upon by using a set of simple, well-defined operations. The REST architectural style constrains an architecture to a client-server architecture, and is designed to use a stateless communication protocol, typically HTTP. In the REST architecture style, clients and servers exchange representations of resources using a standardized interface and protocol. These principles encourage RESTful applications to be simple, lightweight, and have high performance.

RESTful integration

As more and more services become RESTful, the integration challenges also increase. Although RESTful API is something that the world is moving towards, we still have to deal with a lot of legacy systems and services that are not necessarily RESTful. For instance, a given organization may have several SOAP-based web services and some other legacy systems that need to be integrated with a RESTful service or cloud service, such as Twitter switch offering a REST API.

WSO2 ESB foresees rapid growth for REST APIs in the enterprise IT and introduces the REST API component that allows users to create and consume virtual RESTful services. WSO2 API Manager has leveraged this functionality; it addresses real API management requirements, such as versioning, monitoring, monetizing, and life cycle management.

However, this is all about exposing REST APIs from ESB, but how about integrating existing RESTful services? Prior to the 4.7 release, WSO2 ESB had the support to integrate RESTful services, but the users had to do a lot of tweaking to various properties defined in the ESB configuration. Therefore, we introduced a new endpoint type called 'HTTP Endpoint', where users can specify a URI template that can dynamically populate the final URI for the RESTful service invocation. Moreover, users can manipulate the HTTP method of the outgoing request.

HTTP endpoint

HTTP endpoint is a logical representation of an actual resource that allows users to specify the noun and verb in RESTful style. So, we can simply represents a resource with a URI-template and the required HTTP verb can be selected too. The URI-template is fully compliant with RFC 6570[4] spec and the variable names should start with uri.var.* or query.param.*. A given REST API defined in WSO2 ESB can map the URI parameters in to uri.var.* or query.param.* and later can be reused when invoking the actual backend RESTful service (refer to the sample scenario given here).

This is a sample HTTP endpoint representation :

        <endpoint xmlns="http://ws.apache.org/ns/synapse" name="HTTPEndpoint">
            <http uri-template="http://localhost:8080/{uri.var.servicepath}/restapi/{uri.var.servicename}/menu?category={uri.var.category}&amp;type={uri.var.pizzaType}"
            method="GET">
            </http>
        </endpoint>
    

With the introduction of the HTTP endpoint, the use of REST_URL_POSTFIX becomes obsolete and all the tweaks you had to do with such properties can be eliminated in the RESTful integration scenarios.

JSON support for payload factory mediator

When dealing with RESTful services, data formats play an important role. In particular, many RESTful APIs use JSON as the data format and that's what leads us to think about supporting multiple media types in payload factory mediator. Since the payload factory mediator was introduced in ESB, it has been increasingly used for implementing many transformation scenarios (where the transformation logic is one-to-one mapping).

With the ESB 4.7 release, the payload factory mediator supports multiple media types, XML and JSON and caters to the following transformation scenarios. When dealing with multiple media types, we introduce something called an 'evaluator' to distinguish between the xml, json or any other evaluator (i.e. xpath for xml and json_path for json). The media type based transformation does not use any intermediate data representation format, which makes it more efficient.

  • JSON -> JSON
  • JSON -> XML
  • XML -> JSON
  • XML -> XML

We can define the JSON payload as part of the payload factory format and specify the mediation type for the outgoing content. And based on the incoming message type, we can either use XPath or JSON Path as the Payload Factory argument expression. For instance, if the incoming message is JSON, then we can use the JSON path as follows.

   
        <payloadFactory media-type="json">
                 <format>{"purchaseInformation": {"amount": "$1","cc": "$2"}}</format>
                 <args>
                        <arg evaluator="json" expression="$.payment.amount_lkr"></arg>
                        <arg evaluator="json" expression="$.payment.card_no"></arg>
                 </args>
        </payloadFactory>             
    

Native JSON support

JSON is becoming increasingly popular as a message interchange format and many RESTful integration scenarios would need first class support from the integration platform. Prior to WSO2 ESB 4.8, all JSON-based integration scenarios were implemented by using SOAP as an internal canonical message format with in the mediation flows of WSO2 ESB. However, from ESB 4.8 onwards, we offer first class support for JSON, which means there will be no message conversions (back and forth) between an internal canonical form, but you can write the mediation logic based against the incoming JSON payload with no conversion. Native JSON support is allowed in all the basic mediators in WSO2 ESB 4.8, which includes, property, filter, switch and log mediator, etc. The native JSON path evaluation is done with the use of json-eval(json_path_expression). In what follows, we have demonstrated how we can carry out content-based routing using a filter mediator against the incoming JSON without converting the incoming message to a canonical form.

   
     <filter source="json-eval(pizza.name)" regex="Meat Sizzler"> 
	   <then> 
		<log level="custom"> 
			<property name="THEN_FLOW" value="Pizza Found"/> 
		</log> 
	  </then> 
	  <else> 
		<log level="custom"> 
			<property name="ELSE_FLOW" value="Not Found"/> 
		</log> 
	  </else> 
      </filter>
    

Let's see how these new constructs can be applied in real-world RESTful Integration scenarios. For this, we will use the following PizzaShop example.

RESTful integration with WSO2 ESB - PizzaShop scenario

There is a PizzaShop IT system that's implemented as a JAX-RS RESTful service where all the business functionalities are implemented as a RESTful API. PizzaShop wants to expose these business functionalities to its customers and the internal administrators of PizzaShop. In that regard

  • APIs exposed to the customers must have its own API representation and should support JSON as the message format. However, sometimes the JSON format would differ from the format that the backend service requires; therefore, we need to do transformations in some scenarios.
  • The API exposed to the PizzaShop administrator is a SOAP API where we need to send a SOAP message and the integration solution needs to take care of the required transformations, etc.

These are the main business functionalities exposed by their backend.

Pizza menu functionalities

Customers should be able to retrieve the available pizza menu by using either of the following requests.

GET http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/all/

GET http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/?q=pan&type=crust

Testing :

curl -v "http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/all/"

curl -v "http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza/?q=pan&type=crust"

Pizza ordering functionalities

Once the customer selects the required pizza type then he/she can order a given pizza with its ID and with the required quantity.

Add order

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/order

        {"order": {
            "items": [
                {"id": 1, "qty": 2},
                {"id": 2, "qty": 1}
            ]
        }}
    

Get order status

Once the customer places an order, he/she can check the order status as well.

GET http://localhost:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

Update order

Customers can update an existing order as follows.

PUT http://10.111.79.206:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

        {order: {
            items: [
                {id: 1, qty: 5},
                {id: 3, qty: 1}
            ]
        }}
    

Delete order

An existing order can be deleted as well.

DELETE http://localhost:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

Purchasing functionalities

The final state of the customer interaction is the purchasing. The customers can purchase a given order as follows.

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/order/purchase/1379562336828

        { "payment":
            {
                "amount_lkr": "175.00",
                "card_no": "1234-5678-9876-5432"
            }
        }
    

PizzaShop admin functionalities

In the point of view of the PizzaShop admin, they should be able to add any new pizza type into the system. (For simplicity, we have considered only Add operations and omitted update and delete pizza operations)

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/pizza

        { pizza: {
            name: "Meat Sizzler",
            price: 500.0,
            toppings: [
                {
                    id: 9999,
                    name: "Steak",
                    extraPrice: 4.00,
                    category: NONVEG
                },
                {
                    id: 9998,
                    name: "Sun Dried Tomato",
                    extraPrice: 4.00,
                    category: VEGETARIAN
                },
                {
                    id:9997,
                    name: "Mixed Peppers",
                    extraPrice: 3.00,
                    category: VEGETARIAN
                },
                {
                    id: 9996,
                    name: "Cajun Chicken",
                    extraPrice: 3.00,
                    category: NONVEG
                },
                {
                    id: 9995,
                    name: "Chorizo Sausage",
                    extraPrice: 4.00,
                    category: NONVEG
                }
            ]
        }}
    

Testing the backend JAX-RS service:

  • Pre-Req : WSO2 ESB 4.8 M3 or higher, WSO2 AS 5.2.0
  • Change port offsets in carbon.xml as follows : ESB - 0, AS - 1
  • Unzip the pizzashop-rs_1.0.zip in to wso2as-5.2.0/repository/deployment/server/webapps
  • Now the JAX-RS web app should be deployed in AS
  • We need to verify all the about functionalities by directly invoking the web app via curl as shown above
  • Please refer PizzaShop_test.txt for all the associated request and response.

Solution architecture

We can suggest the following solution architecture to implement the integration solution for PizzaShop. Here, we have used WSO2 ESB as the integration platform and we have used REST APIs of WSO2 ESB to expose a RESTful API from ESB to PizzaShop customers. For the PizzaShop admin API we can use a proxy service in the ESB to expose a SOAP web service to the PizzaShop application (JAX-RS is hosted inside WSO2 AS, but we can use any application server too as the backend)

PizzaShop solution architecture

Implementation of PizzaShop scenario

Designing the customer API

In order to design the customer API, we can identify the resources associated with the scenario. The following resources can be defined with the required URI-templates.

API : pizzashop : /pizzashop
        Pizza - /api/menu/pizza*       		[GET]
        Order - /api/order*
        Order - /api/order/{orderId}       		[POST, GET, DELETE, PUT]
        Purchase - /api/purchase/{orderId}		[POST]
    

Let's discuss how the message flows can be defined against these resources and how we leverage HTTP endpoint when invoking an external RESTful service.

Resource : /api/menu/pizza

For this resource, we only allow GET calls from this API. However, there can be two types of GET requests coming through to retrieve the pizza menu.

http://127.0.0.1:8280/pizzashop/api/menu/pizza

http://127.0.0.1:8280/pizzashop/api/menu/pizza?val=thin&type=crust

For this resource, the URI template can be defined as ‘/api/menu/pizza*” and from the API itself, the query parameters that come as part of the request are evaluated and populated into the context as variables stars with ‘query.param.*’. Hence they can be used to distinguish the request with the query parameters and the request that doesn’t (filter mediator). Then we are all set to create the HTTP endpoint for both these cases. For the request with query parameters, the HTTP EP URI template will be ‘http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza?q={query.param.val}&type={query.param.type}’.

        <resource methods="GET" uri-template="/api/menu/pizza*">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Info API - IN"></property>
                    <property name="Type" expression="$ctx:query.param.type"></property>
                    <property name="Val" expression="$ctx:query.param.val"></property>
                </log>
                <filter xpath="$ctx:query.param.type and $ctx:query.param.val">
                    <then>
                        <send>
                            <endpoint>
                                <http method="get" uri-template="http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza?q={query.param.val}&type={query.param.type}"></http>
                            </endpoint>
                        </send>
                    </then>
                    <else>
                        <send>
                            <endpoint>
                                <http method="get" uri-template="http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza/all"></http>
                            </endpoint>
                        </send>
                    </else>
                </filter>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Resource : /api/order* and /api/order/{orderId}

The orders are subject to all four CRUD operations (POST, PUT, GET, DELETE) and in this case the customer application is using the same message format (JSON) as the backend service. Hence, no transformation is required.

For the request with the order ID (i.e. get order status, update order, delete order), the URL template /api/order/{orderId} is used and the matching order ID is stored in the context per each message in the form of url.var.*. (eg: uri.var.orderId) and this can be directly used when invoking the backend RESTful service with HTTP EP.

eg: http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}

        <resource methods="POST" uri-template="/api/order*">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Order API - IN"></property>
                </log>
                <send>
                    <endpoint>
                        <http method="post" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order"></http>
                    </endpoint>
                </send>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
        <resource methods="POST GET DELETE PUT" uri-template="/api/order/{orderId}">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Order API - IN"></property>
                </log>
                <switch source="$axis2:HTTP_METHOD">
                    <case regex="GET">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order GET ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="GET" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                    <case regex="PUT">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order PUT ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="PUT" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                    <case regex="DELETE">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order DELETE ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="DELETE" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                </switch>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Resource - /api/purchase/{orderId}

This is very similar to order resource, because it uses orderId for each purchase request. However, unlike all above scenarios, in this scenario the incoming JSON message format is different from the request format. So, we have to transform the incoming JSON message to the outgoing JSON message format with the use of JSON Payload Factory.

incoming format :

        { "payment":
            {
                "amount_lkr": "175.00",
                "card_no": "1234-5678-9876-5432"
            }
        }
    

outgoing format :

        {purchaseInformation:
            {
                amount: 175.00,
                cc: "1234-5678-9876-5432"
            }
        }
    

With the ESB 4.7 or higher, we support JSON transformations with payload factory mediator. In order to select the required nodes from the incoming message, we support JSON path representations, which is defined in http://goessner.net/articles/JsonPath/.

        <resource methods="POST" uri-template="/api/purchase/{orderId}">
            <inSequence>
                    <payloadFactory media-type="json">
                        <format>{"purchaseInformation": {"amount": "$1","cc": "$2"}}</format>
                        <args>
                            <arg evaluator="json" expression="$.payment.amount_lkr"></arg>
                            <arg evaluator="json" expression="$.payment.card_no"></arg>
                        </args>
                    </payloadFactory>
                    <send>
                        <endpoint>
                            <http method="post" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/purchase/{uri.var.orderId}"></http>
                        </endpoint>
                    </send>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Designing the PizzaShop admin API

As discussed above, the main functionality offered from PizzaShop admin API is adding new pizza type to the system. Unlike APIs, this needs to be a SOAP web service. Hence, we can design this as an ESB proxy service. This use case leads us to use payload factory for SOAP ->JSON conversion; let's see how it can be done.

Let's assume that we defined the incoming message format as follows:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <pizza>
         <name>Meat Sizzler</name>
         <price>500.0</price>
         <toppings>
            <topping>
               <id>9999</id>
               <name>Steak</name>
               <extraPrice>4.00</extraPrice>
               <category>NONVEG</category>
            </topping>
            <topping>
               <id>9998</id>
               <name>Sun Dried Tomato</name>
               <extraPrice>4.00</extraPrice>
               <category>VEGETARIAN</category>
            </topping>
            <topping>
               <id>9997</id>
               <name>Mixed Peppers</name>
               <extraPrice>3.00</extraPrice>
               <category>VEGETARIAN</category>
            </topping>
            <topping>
               <id>9996</id>
               <name>Cajun Chicken</name>
               <extraPrice>3.00</extraPrice>
               <category>NONVEG</category>
            </topping>
            <topping>
               <id>9995</id>
               <name>Chorizo Sausage</name>
               <extraPrice>4.00</extraPrice>
               <category>NONVEG</category>
            </topping>
         </toppings>
      </pizza>
   </soapenv:Body>
</soapenv:Envelope>
    

Required JSON request

       {"pizza": {
          "name": "Meat Sizzler",
          "price": "500.0",
          "pf_ghost_field1": {
          "topping": [
            {
                "id": "9999",
                "name": "Steak",
                "extraPrice": "4.00",
                "category": "NONVEG"
            },
            {
                "id": "9998",
                "name": "Sun Dried Tomato",
                "extraPrice": "4.00",
                "category": "VEGETARIAN"
            },
            {
                "id": "9997",
                "name": "Mixed Peppers",
                "extraPrice": "3.00",
                "category": "VEGETARIAN"
            },
            {
                "id": "9996",
                "name": "Cajun Chicken",
                "extraPrice": "3.00",
                "category": "NONVEG"
            },
            {
                "id": "9995",
                "name": "Chorizo Sausage",
                "extraPrice": "4.00",
                "category": "NONVEG"
            }
        ]
    }
}}
    

SOAP - JSON transformation with payload factory

We can use JSON payload factory to create the required JSON format out of the incoming XML format. However, if we carefully observe the incoming XML and out going JSON message, we could see that it's not just one-to-one mapping (i.e. toppings tag in XML and toppings JSON array). Therefore, we can evaluate the required parameters of the incoming SOAP message and add that into the new JSON payload. To handle exceptional cases, such as how topping list is mapped into a JSON array, we can mix and match Xpath and JSON path as follows.

<proxy xmlns="http://ws.apache.org/ns/synapse" name="PizzaAdminService" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">
        	<target>
        		<inSequence>
        			<payloadFactory media-type="json">
            <format>{ "pizza": {    "name": "$1",    "price": $2,    "topping": $3}}</format>
                            <args>
                                <arg evaluator="xml" expression="//pizza/name"/>
                                <arg evaluator="xml" expression="//pizza/price"/>
                                <arg evaluator="json" expression="pizza.toppings.topping"/>
                            </args>
</payloadFactory>
        			<property name="messageType" value="application/json" scope="axis2"/>
        			<send>
        				<endpoint>
        					<http method="post"
        					uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/pizza"/>
        				</endpoint>
        			</send>
        		</inSequence>
        		<outSequence>
        			<send/>
        		</outSequence>
        	</target>
        	<description/>
</proxy>
    

The PizzaAdminService can be invoked by sending the above SOAP request to the proxy service: http://localhost:8280/services/PizzaAdminService

Resources

All required config and docs can be found at https://github.com/kasun04/dist/tree/master/scenarios/pizzashop

Conclusion

Integrating existing RESTful service is a common requirement in modern enterprises as we are increasingly consuming RESTful API. WSO2 ESB introduces a rich set of features to cater to the RESTful integration scenarios. HTTP endpoint and payload factory can be used to implement any RESTful integration scenario and this article provides a complete overview of the use cases related to HTTP endpoint and JSON Payload Factory.

References

  1. Representational State Transfer (REST)
  2. Is REST Successful in the Enterprise?
  3. RESTful Service Best Practices
  4. URI Template