WSO2Con 2013 CFP Banner

RESTful Integration with WSO2 ESB

Discuss this article on Stack Overflow
By Kasun Indrasiri
  • 21 Jan, 2014
  • Level: Advanced
  • Reads: 8272

With the increasing usage of RESTful APIs, any integration platform needs cater the real worlds RESTful integration needs. This articles gives you a comprehensive understanding of how WSO2 ESB can be used to realise all the RESTful integration needs. In particular, this article targets how resources and associated operations (HTTP Verbs) can be integration with HTTP Endpoint and the usage of Payload Factory with JSON support for JSON-JSON and XML-JSON transformation scenarios.

kasun's picture
Kasun Indrasiri
Architect
WSO2 Inc

Applies To

WSO2 ESB 4.8.0
WSO2 AS 5.2.1

Table of Content

Introduction

Representational State Transfer (REST) increasingly becoming popular over the Web as a simpler alternative to SOAP web services. It was initially introduced by Roy Fielding as "Architectural Styles and the Design of Network-based Software Architectures"[1] which analyzes a set of software architecture principles that use the Web as a platform for distributed computing. Since then, it was heavily adopted into most of the web based software solutions and its pretty much supersedes SOAP in most web services based solutions.

source : Is REST Successful in the Enterprise? [2]

REST is an architecture style for designing networked applications or distributed hypermedia systems based on the Resource Oriented Architecture(ROA) paradigm.

REST ignores the details of component implementation and protocol syntax in order to focus on the roles of components, the constraints upon their interaction with other components, and their interpretation of significant data elements.

REST Triangle

REST Triangle [3]

Nouns - Identification and addressability of Resources

  • A 'resource' is a key abstraction REST and any information that can be named can be a resource.
  • Everything that service provides is a resource. For instance, a text file, document, image, a collection of other resources, a non-virtual object can be considered as resource. So, such interesting bits of information are identified with URIs and are usually accessed via URL
  • Everything of interest should be named.
  • Separating nouns from verbs and representations.
  • Resources are defined by URIs :
  • http://example.com/customers/1234

    http://example.com/orders/2007/10/776654

    http://example.com/products/4554

    http://example.com/processes/salary-increase-234

Verbs - Operations that can be applied on a Resources/CRUD

Verbs are generally used as operations, generally encompasses everything that can be done to a resource (or a piece of data). Mostly CRUD (Create, Read, Update, Delete) operations. In HTTP scope these are mapped to POST, GET, PUT and DELETE along with some additional verbs such as OPTIONS and HEAD.

  • GET : The HTTP GET method is used to retrieve a representation of a resource.
  • POST: The POST verb is most-often utilized for creation of new resources.
  • PUT : PUT is most-often utilized for update capabilities, PUT-ing to a known resource URI with the request request body containing the newly-updated representation of the original resource.
  • DELETE : Delete a resource identified by a URI.
  • HEAD : Similar to GET but a response for a HEAD request must not have a body, though the meta-information are identical to a response to a GET request for the same resource uri. This method is often used for testing hypertext links for validity, accessibility, and recent modification.
  • OPTIONS: This represents a request for information about the communication options available on the request/response chain identified by the Request-URI

Data Formats - Identification and addressability of Resources

Data Formats provide the format for the data that will take part in your RESTful discussion. Most commons data types are XML JSON, or CSV. A resource, or resource state, is the data that defines the resource representation. So, for requests such as POST or PUT should contain the resource representation as the content body (or entity body). When it comes to returning representations, most REST APIs allows on demand data representation, in which users can request the required returning representation. Once could possibly use HTTP ‘Accept’ header to specify the required returning representation. However, most of the modern REST APIs uses format specifiers, which looks more like a file extension. For instance GET http://www.example.com/customers.xml will return an xml representation while GET http://www.example.com/customers.json gives a json response representation.

In a nutshell, the REST architectural style, data and functionality are considered resources, and these resources are accessed using Uniform Resource Identifiers (URIs), typically links on the web. The resources are acted upon by using a set of simple, well-defined operations. The REST architectural style constrains an architecture to a client-server architecture, and is designed to use a stateless communication protocol, typically HTTP. In the REST architecture style, clients and servers exchange representations of resources using a standardized interface and protocol. These principles encourages RESTful applications to be simple, lightweight, and have high performance.

RESTful Integration

As more and more services becoming RESTful, the integration challenges also increases. Although, RESTful API is something that the world is moving towards, but we still have to deal with a lot of legacy systems and services which are not necessary RESTful. For instance, a given organization may have several SOAP based web services and some other legacy systems that need to be integrated with a RESTful service or cloud service such as twitter switch offers a REST API.

WSO2 ESB foresees the rapid growth of REST APIs in the enterprise IT and introduces the REST API component which allows the uses to create and consume virtual RESTful services. WSO2 API Manager leverage this functionality and took several steps further by introducing WSO2 API Manager, which address the real API Management requirements such as versioning, monitoring, monetizing and life cycle management.

However, this is all about exposing REST APIs from ESB, but how about integrating existing RESTful services? Prior to 4.7 release, WSO2 ESB has the support for integrating RESTful services but the users had to do a lot of tweak around various properties defined in the ESB configuration. Therefore, we introduced a new endpoint type called 'HTTP Endpoint', where users can specify an URI Template which can dynamically populate final URI for the RESTful service invocation. Also, users can manipulate HTTP method of the outgoing request.

HTTP Endpoint

HTTP Endpoint is a logical representation of an actual resource which allows the users to specify the noun and verb in RESTful style. So, we can simply represents a resource with a uri-template and the required HTTP Verb can be selected too. The URI-Template is fully compliant with RFC 6570[4] spec and the variable names should start with uri.var.* or query.param.*. A given REST API defined in WSO2 ESB can map the uri parameters in to uri.var.* or query.param.* and later can be reused when invoking the actual backend RESTful service (Please refer the sample scenario given here).

This is a sample HTTP Endpoint representation :

        <endpoint xmlns="http://ws.apache.org/ns/synapse" name="HTTPEndpoint">
            <http uri-template="http://localhost:8080/{uri.var.servicepath}/restapi/{uri.var.servicename}/menu?category={uri.var.category}&amp;type={uri.var.pizzaType}"
            method="GET">
            </http>
        </endpoint>
    

With the introduction of the HTTP Endpoint, the use of REST_URL_POSTFIX becomes obsolete and all the tweaks you had to do with such properties can be eliminated in the RESTful Integration scenarios.

JSON support for Payload Factory Mediator

When dealing with RESTful services, the data formats plays an important role. In particular, many RESTful APIs uses JSON as the data format and that's what leads us to think about supporting multiple media types in Payload Factory mediator. Since the Payload Factory mediator was introduced in to ESB, it was increasingly used for implementing many transformation scenarios (where the transformation logic is one to one mapping).

With ESB 4.7 release, the payload factory mediator support multiple media types, xml and json and carter following transformation scenarios. When dealing with multiple media types, we introduce something called an 'evaluator' to distinguish between the xml, json or any other evaluator(i.e. xpath for xml and json_path for json). The media type based transformation not using any intermediate data representation format which makes it more efficient.

  • JSON -> JSON
  • JSON -> XML
  • XML -> JSON
  • XML -> XML

We can define the JSON payload as part of the payload factory format and specify the mediation type for the outgoing content. And based on the incoming message type, we can either use XPath or JSON Path as the Payload Factory argument expression. For instance, if the incoming message is JSON, then we can use the JSON path as follows.

   
        <payloadFactory media-type="json">
                 <format>{"purchaseInformation": {"amount": "$1","cc": "$2"}}</format>
                 <args>
                        <arg evaluator="json" expression="$.payment.amount_lkr"></arg>
                        <arg evaluator="json" expression="$.payment.card_no"></arg>
                 </args>
        </payloadFactory>             
    

Native JSON Support

JSON is increasing getting popular as a message interchange format and many RESTful integration scenarios would need the first class support from the integration platform. Prior to WSO2 ESB 4.8, all JSON based integration scenarios were implemented by using SOAP as an internal canonical message format with in the mediation flows of WSO2 ESB. However, from ESB 4.8 onwards we offer the first class support for JSON, which means there will be no message conversion happens back and forth between an internal canonical form but you can write the mediation logic based against the incoming JSON payload with no conversion. Native JSON support is allowed in all the basic mediators in WSO2 ESB 4.8 which includes, property, filter, switch and log mediator etc. Native JSON path evaluation is done with the use of json-eval(json_path_expression). In what follows, we have demonstrated how we can do content based routing using a filter mediator against the incoming JSON with out converting the incoming message to a canonical form.

   
     <filter source="json-eval(pizza.name)" regex="Meat Sizzler"> 
	   <then> 
		<log level="custom"> 
			<property name="THEN_FLOW" value="Pizza Found"/> 
		</log> 
	  </then> 
	  <else> 
		<log level="custom"> 
			<property name="ELSE_FLOW" value="Not Found"/> 
		</log> 
	  </else> 
      </filter>
    

Lets see how these new constructs can be applied in real world RESTful Integration scenarios. For that we will use the following PizzaShop example.

RESTful Integration with WSO2 ESB - PizzaShop Scenario

There is a PizzaShop IT system which is implemented as a JAX-RS RESTful service where all the business functionalities are implemented as a RESTful API. The PizzaShop wants to expose these business functionalities to its customers and the internal administrators of PizzaShop. In that regard

  • APIs exposed to the customers must have its own API representation and it supports JSON as the message format. However, sometimes the JSON format differ from the format that the backend service requires and therefore we need to do transformations in some scenarios.
  • The API exposed to the PizzaShop adminstrator is a SOAP API where we need to send a SOAP message and the integration solution needs to take care of required transformations etc.

These are the main business functionalities exposed by their backend.

Pizza Menu Functionalities

The customers should be able to retrieve the available pizza menu by using either of the following request.

GET http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/all/

GET http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/?q=pan&type=crust

Testing :

curl -v "http://127.0.0.1:9764/pizzashop_jaxrs-1.0/services/menu/pizza/all/"

curl -v "http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza/?q=pan&type=crust"

Pizza Ordering Functionalities

Once the customer selects the required pizza type then he can order a given pizza with its Id and with the required quantity.

Add Order

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/order

        {"order": {
            "items": [
                {"id": 1, "qty": 2},
                {"id": 2, "qty": 1}
            ]
        }}
    

Get Order Status

Once the customer place an order, he can check the order status as well.

GET http://localhost:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

Update Order

Customers can update an existing order as follows.

PUT http://10.111.79.206:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

        {order: {
            items: [
                {id: 1, qty: 5},
                {id: 3, qty: 1}
            ]
        }}
    

Delete Order

An existing order can be deleted as well.

DELETE http://localhost:9764/pizzashop-rs_1.0/services/menu/order/1379562336828

Purchasing Functionalities

The final state of the customer interaction is the purchasing. The customers can purchase a given order as follows.

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/order/purchase/1379562336828

        { "payment":
            {
                "amount_lkr": "175.00",
                "card_no": "1234-5678-9876-5432"
            }
        }
    

PizzaShop Admin Functionalities

In the point of view of the PizzaShop admin, he should be able to add any new pizza type in to the system. (For simplicity, we have considered only Add operations and omit update and delete pizza operations)

POST http://localhost:9764/pizzashop-rs_1.0/services/menu/pizza

        { pizza: {
            name: "Meat Sizzler",
            price: 500.0,
            toppings: [
                {
                    id: 9999,
                    name: "Steak",
                    extraPrice: 4.00,
                    category: NONVEG
                },
                {
                    id: 9998,
                    name: "Sun Dried Tomato",
                    extraPrice: 4.00,
                    category: VEGETARIAN
                },
                {
                    id:9997,
                    name: "Mixed Peppers",
                    extraPrice: 3.00,
                    category: VEGETARIAN
                },
                {
                    id: 9996,
                    name: "Cajun Chicken",
                    extraPrice: 3.00,
                    category: NONVEG
                },
                {
                    id: 9995,
                    name: "Chorizo Sausage",
                    extraPrice: 4.00,
                    category: NONVEG
                }
            ]
        }}
    

Testing the backend JAX-RS service:

  • Pre-Req : WSO2 ESB 4.8 M3 or higher, WSO2 AS 5.2.0
  • Change port offsets in carbon.xml as follows : ESB - 0, AS - 1
  • Unzip the pizzashop-rs_1.0.zip in to wso2as-5.2.0/repository/deployment/server/webapps
  • Now the JAX-RS web app should be deployed in AS
  • We need to verify all the about functionalities by directly invoking the web app via curl as shown above
  • Please refer PizzaShop_test.txt for all the associated request and response.

Solution Architecture

We can suggest the following solution architecture to implement the integration solution for the PizzaShop. Here we have used WSO2 ESB as the integration platform and we have used REST APIs of WSO2 ESB to expose a RESTful API from ESB to the PizzaShop customers. For the PizzaShop Admin API we can use Proxy Service in ESB to expose a SOAP web service to the PizzaShop application. (JAX-RS is hosted inside WSO2 AS, but we can use any application server to as the backend)

PizzaShop Solution Architecture

Implementation of PizzaShop Scenario

Designing the Customer API

In order to design the customer API, we can identify the resources associated with the scenario. Following, resources can be defined with the required uri-templates.

API : pizzashop : /pizzashop
        Pizza - /api/menu/pizza*       		[GET]
        Order - /api/order*
        Order - /api/order/{orderId}       		[POST, GET, DELETE, PUT]
        Purchase - /api/purchase/{orderId}		[POST]
    

Lets discuss how the message flows can be defined against these resources and how do we leverage HTTP Endpoint when invoking an external RESTful service.

Resource : /api/menu/pizza

For this resource, we only allow GET calls from this API. However, there can be two type of GET request coming through to retrieve the pizza menu.

http://127.0.0.1:8280/pizzashop/api/menu/pizza

http://127.0.0.1:8280/pizzashop/api/menu/pizza?val=thin&type=crust

For this resource, uri template can be defined as ‘/api/menu/pizza*” and from the API itself, the query parameters that are coming as part of the request are evaluated and populated into the context as variables stars with ‘query.param.*’. Hence they can be used to distinguish the request with the query params and the request that doesn’t (filter mediator). Then we are all set to create the HTTP Endpoint for both these cases. For the request with query params, HTTP EP uri template will be ‘http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza?q={query.param.val}&type={query.param.type}’.

        <resource methods="GET" uri-template="/api/menu/pizza*">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Info API - IN"></property>
                    <property name="Type" expression="$ctx:query.param.type"></property>
                    <property name="Val" expression="$ctx:query.param.val"></property>
                </log>
                <filter xpath="$ctx:query.param.type and $ctx:query.param.val">
                    <then>
                        <send>
                            <endpoint>
                                <http method="get" uri-template="http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza?q={query.param.val}&type={query.param.type}"></http>
                            </endpoint>
                        </send>
                    </then>
                    <else>
                        <send>
                            <endpoint>
                                <http method="get" uri-template="http://127.0.0.1:9764/pizzashop-rs_1.0/services/menu/pizza/all"></http>
                            </endpoint>
                        </send>
                    </else>
                </filter>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Resource : /api/order* and /api/order/{orderId}

The orders are subjected to all four CRUD operations (POST, PUT, GET, DELETE) and in this case the customer application is using the same message format (JSON) as the backend service. Hence no transformation is required.

For the request with the order ID (ie: get order status, update order, delete order), the url template /api/order/{orderId} is used and the matching order ID is stored in the context per each message in the form of url.var.*. (eg: uri.var.orderId) and this can be directly use when invoking the backend RESTful service with HTTP EP.

eg: http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}

        <resource methods="POST" uri-template="/api/order*">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Order API - IN"></property>
                </log>
                <send>
                    <endpoint>
                        <http method="post" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order"></http>
                    </endpoint>
                </send>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
        <resource methods="POST GET DELETE PUT" uri-template="/api/order/{orderId}">
            <inSequence>
                <log>
                    <property name="Message Flow" value="Pizza Order API - IN"></property>
                </log>
                <switch source="$axis2:HTTP_METHOD">
                    <case regex="GET">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order GET ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="GET" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                    <case regex="PUT">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order PUT ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="PUT" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                    <case regex="DELETE">
                        <log level="custom">
                            <property name="Message Flow" value="--- Order DELETE ---"></property>
                        </log>
                        <send>
                            <endpoint>
                                <http method="DELETE" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/{uri.var.orderId}"></http>
                            </endpoint>
                        </send>
                    </case>
                </switch>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Resource - /api/purchase/{orderId}

This is very similar to order resource, because it uses orderId for each purchase request. However, unlike all above scenarios, in this scenario the incoming JSON message format is different from the request format. So, we have to transform the incoming json message to the outgoing json message format with the use of JSON Payload Factory.

incoming format :

        { "payment":
            {
                "amount_lkr": "175.00",
                "card_no": "1234-5678-9876-5432"
            }
        }
    

outgoing format :

        {purchaseInformation:
            {
                amount: 175.00,
                cc: "1234-5678-9876-5432"
            }
        }
    

With the ESB 4.7 or higher, we do support JSON transformations with Payload Factory mediator. In order to select the required nodes from the incoming message we support JSON Path representations which is defined in http://goessner.net/articles/JsonPath/.

        <resource methods="POST" uri-template="/api/purchase/{orderId}">
            <inSequence>
                    <payloadFactory media-type="json">
                        <format>{"purchaseInformation": {"amount": "$1","cc": "$2"}}</format>
                        <args>
                            <arg evaluator="json" expression="$.payment.amount_lkr"></arg>
                            <arg evaluator="json" expression="$.payment.card_no"></arg>
                        </args>
                    </payloadFactory>
                    <send>
                        <endpoint>
                            <http method="post" uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/order/purchase/{uri.var.orderId}"></http>
                        </endpoint>
                    </send>
            </inSequence>
            <outSequence>
                <send></send>
            </outSequence>
        </resource>
    

Designing the PizzaShop Admin API

As discussed above, the main functionality offered from PizzaShop admin API is that adding new pizza type to the system. Unlike about APIs, this needs to be a SOAP web service. Hence we can design this as an ESB proxy service. This use case leads us to use Payload factory for SOAP ->JSON conversion and let see how it can be done.

Lets assume that we defined the incoming message format as follows:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <pizza>
         <name>Meat Sizzler</name>
         <price>500.0</price>
         <toppings>
            <topping>
               <id>9999</id>
               <name>Steak</name>
               <extraPrice>4.00</extraPrice>
               <category>NONVEG</category>
            </topping>
            <topping>
               <id>9998</id>
               <name>Sun Dried Tomato</name>
               <extraPrice>4.00</extraPrice>
               <category>VEGETARIAN</category>
            </topping>
            <topping>
               <id>9997</id>
               <name>Mixed Peppers</name>
               <extraPrice>3.00</extraPrice>
               <category>VEGETARIAN</category>
            </topping>
            <topping>
               <id>9996</id>
               <name>Cajun Chicken</name>
               <extraPrice>3.00</extraPrice>
               <category>NONVEG</category>
            </topping>
            <topping>
               <id>9995</id>
               <name>Chorizo Sausage</name>
               <extraPrice>4.00</extraPrice>
               <category>NONVEG</category>
            </topping>
         </toppings>
      </pizza>
   </soapenv:Body>
</soapenv:Envelope>
    

Required JSON Request

       {"pizza": {
          "name": "Meat Sizzler",
          "price": "500.0",
          "pf_ghost_field1": {
          "topping": [
            {
                "id": "9999",
                "name": "Steak",
                "extraPrice": "4.00",
                "category": "NONVEG"
            },
            {
                "id": "9998",
                "name": "Sun Dried Tomato",
                "extraPrice": "4.00",
                "category": "VEGETARIAN"
            },
            {
                "id": "9997",
                "name": "Mixed Peppers",
                "extraPrice": "3.00",
                "category": "VEGETARIAN"
            },
            {
                "id": "9996",
                "name": "Cajun Chicken",
                "extraPrice": "3.00",
                "category": "NONVEG"
            },
            {
                "id": "9995",
                "name": "Chorizo Sausage",
                "extraPrice": "4.00",
                "category": "NONVEG"
            }
        ]
    }
}}
    

SOAP - JSON Transformation with Payload Factory

We can use JSON Payload Factory to create the required JSON format out of the incoming XML format. However, if we carefully observe the incoming xml and out going JSON message, we could see that its not just one to one mapping (i.e. toppings tag in XML and toppings JSON array). So, we can evaluate the required parameters from the incoming SOAP message and add that into the new JSON payload. To handle exceptional cases such as how topping list is mapped in to a JSON array, we can mix and match xpath and json path as follows.

<proxy xmlns="http://ws.apache.org/ns/synapse" name="PizzaAdminService" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">
        	<target>
        		<inSequence>
        			<payloadFactory media-type="json">
            <format>{ "pizza": {    "name": "$1",    "price": $2,    "topping": $3}}</format>
                            <args>
                                <arg evaluator="xml" expression="//pizza/name"/>
                                <arg evaluator="xml" expression="//pizza/price"/>
                                <arg evaluator="json" expression="pizza.toppings.topping"/>
                            </args>
</payloadFactory>
        			<property name="messageType" value="application/json" scope="axis2"/>
        			<send>
        				<endpoint>
        					<http method="post"
        					uri-template="http://localhost:9764/pizzashop-rs_1.0/services/menu/pizza"/>
        				</endpoint>
        			</send>
        		</inSequence>
        		<outSequence>
        			<send/>
        		</outSequence>
        	</target>
        	<description/>
</proxy>
    

The PizzaAdminService can be invoked by sending the above SOAP request to the proxy service : http://localhost:8280/services/PizzaAdminService

Resources

All required config and docs can be found at : https://github.com/kasun04/dist/tree/master/scenarios/pizzashop

Conclusion

Integrating existing RESTful service is a common requirement in modern enterprises as we are more and more consuming RESTful API. WSO2 ESB introduces a rich set of features to cater the RESTful integration scenarios. HTTP Endpoint and Payload Factory can be used to implement any RESTful integration scenario and this article provide a complete overview of the use cases related to HTTP Endpoint and JSON Payload Factory.

References

  1. Representational State Transfer (REST)
  2. Is REST Successful in the Enterprise?
  3. RESTful Service Best Practices
  4. URI Template

Author

Kasun Indrasiri, Software Architect, WSO2 Inc

WSO2Con 2014 USA