[Article] WSO2 Data Mapper: Message Transformation Revolutionized

Introduction

Message transformation and translation is a key functionality of enterprise integration. While various legacy systems can be brought together, different interpretations need to be mapped or altered to match the involved entities for clients to work as a single unit.

For example let’s take a business need where several legacy systems such as billing, registration, and order processing must work together to deliver a one-stop experience for the client that spans from a simple registration, placing an order, and at the end of the process generating the billing amount. On the surface what the client sees is a smooth flow of events that fulfills a specific requirement; however, beneath all of this is a web of operations that take place to integrate various legacy systems to work as a single unit.

Figure 1: One-stop experience for the end user

Figure 2: High-level communication with message transformation

For the integration that takes place, message translation and transformation is a basic requirement as legacy systems are designed in different ways. When the various legacy systems are taken into account they might vary in terms of implementation, input channeling, and output channeling. Therefore, when two legacy systems need to work together it’s hard to implement a one-to-one mapping due to the above-mentioned differences.

In such an instance, middleware plays an important role; products such as WSO2 Enterprise Service Bus (ESB) provide solutions to integrate various disparate systems by managing the web of connections, including the very basic message translation and message transformation.

WSO2 ESB has been providing various mechanisms for message translation and transformation processes in the form of mediators, such as Payload Factory, Enrich, XSLT, and Smooks, etc. for a long time. Now, we’re taking one step forward to provide users with a mediator that comes with an attractive graphical interface capability using the WSO2 ESB tooling functionality, namely WSO2 ESB data mapper mediator.

Introducing WSO2 Data Mapper Mediator

Data mapper mediator is a solution that’s provided to support message transformation and message enriching since it can change the message entirely or change the structure of the message. WSO2 ESB provides a tooling component based on WSO2 Developer Studio that enables visual creation of mapping configurations. These configurations are used by the WSO2 data mapping engine to create files that are required for the relevant execution of either message transformation or message enriching.

Figure 3: Data mapper mediator within sequence; illustration with WSO2 ESB tooling

At present, conversion to and from for XML, JSON and CSV is supported by the WSO2 data mapper mediator. It also provides a collection of operators powered by JavaScript that will enable message alterations to occur without much coding hassle for the user. There are string, numerical, logical, type conversion, etc. operators that can cater to many needs of message alterations and transformations. Additionally, the input (inflow of the mediator) and the output (outflow from the mediator) of the data mapper mediator can be defined by either using XML, JSON, XSD, JSONSCHEMA or CONNECTOR types. This has provided further extensibility in message transformation as it provides a one-stop solution for various tasks that had to be done separately.

Figure 4: Canvas area of data mapper with input, output and operators (illustration with WSO2 ESB tooling)

Sample use cases: Usages of WSO2 Data Mapper mediator

Use case 1: Direct message translation and transformation

Taz Airlines currently has legacy backends for various tasks, such as route information retrieval, availability checking, and reservation and cancellation. To move to the next level, the airline has introduced a system that enables the user to obtain information, make reservations, and perform cancellations as a one-stop capability. The legacy systems support Soap messages while the client invocations can be on Soap, JSON, etc.

Message transformation: Requesting route information

Taz Airlines’ web application is designed in such a way where the client needs to only submit the required information once. Initially, the user submits all required information so there’s minimum effort for the user.

Figure 5: The Information mismatch of the input and expected from the legacy system

As shown above, the request contains additional fields than are required by the backend so we need to transform the message.

By using the data mapper mediator, the above transformation can be achieved easily. In order to make this transformation happen through data mapper, the input and the output should be configured. The input is the message that’s received by the data mapper while the output is the result of the data mapper. The input and output is configurable through a predefined template.

Sample input in XML format:

<request>
<name>test</name>
<address>test</address>
<departureDate>0.0.0.0</departureDate>
<placeOfDeparture>test</placeOfDeparture>
<destination>test</destination>
<totalNoOfPeople>1</totalNoOfPeople>
<totalNoOfAdults>1</totalNoOfAdults>
<totalNoOfChildren exists="true">1</totalNoOfChildren>
<totalNoOfPets exists="true">1</totalNoOfPets>
</request>

In the above request, the totalNoOfChildren and the totalNoOfPets can be optional depending on the requestor. Hence, there should be a capability to support nullable fields.

Sample output in XML format:

<routeRequest>
<departureDate>0.0.0.0</departureDate>
<placeOfDeparture>test</placeOfDeparture>
<destination>test</destination>
</routeRequest>

Data mapper diagram that supports the above requirement with the given input and output.

Figure 6: Input and output mapping of data mapper to generate the desired output

As depicted in Figure 6, using the data mapper, a direct translation as required can be done easily with a user-friendly graphical interface. In addition, data mapper includes support for nullable fields so that totalNoOfChildren and the totalNoOfPets being null is enabled.

Use case 2: Message transformation with numerical, conditional operations

TradeZit Markets is conducting a promotion program and they check purchases made throughout a day for one month to understand which products have more sales and those that don’t. The purchases of the day are processed and thereafter the product name and total amount earned are inserted to a database of products that exceed a certain sales limit.

Figure 7: Required data by the data service is mismatched to what is given by the system

Data mapper mediator is not just a tool that can be used for message translation, but it can also be used for advanced message transformations as it consists of operations that can be used for message transformation. It consists of numeric, boolean, string, logical, etc. operators designed in an easy-to-use graphical manner that can support many transformations that had to be written by hand earlier.

In the above scenario, we need to transform the incoming message using the data mapper. Here, we have the capability of using string and numerical operations to process as required. The operations are based on JavaScript language and just by simply dragging and dropping we can easily configure the operations as needed to cater to the individual need.

Sample input in XML format:

<Products>
   <product>
 	<name>test</name>
 	<unitPrice>100.0</unitPrice>
 	<noOfUnits>260</noOfUnits>
   </product>
   <product>
 	<name>test</name>
 	<unitPrice>100.0</unitPrice>
 	<noOfUnits>200</noOfUnits>
   </product>
   <product>
 	<name>test</name>
 	<unitPrice>100.0</unitPrice>
 	<noOfUnits>290</noOfUnits>
   </product>
 </Products>

Sample Output in JSON format:

{
  "Products": {
	"product": [
  	{
    	"name": "test",
    	"totalSales": "100.0"
  	},
  	{
    	"name": "test",
    	"totalSales": "100.0"
  	},
  	{
    	"name": "test",
    	"totalSales": "100.0"
  	}
	]
  }
}

Data mapper diagram that supports the above requirement with the above given input and output.

Figure 8: Input and output mapping of data mapper to generate the desired output

The above mapping will obtain the total sales price of a certain product and compares it against a predefined minimum sales amount; thereafter, if the total sales price exceeds the minimum limit the value is mapped to the output or else a constant value is mapped to represent the product that has not reached the minimum expected sales amount.

Use case 3: Message transformation with logical, string and conditional operations

Smart Life Insurance company is introducing a new insurance policy where the clients can subscribe to the policy. An eligibility check is carried out before subscription. The eligibility check is done by evaluating the patient’s age, nationality, income, medical conditions, and criminal records. The age, nationality, gross income, and medical status information is obtained by the client while the criminal record information is obtained by a search in the legal authority database that’s available to insurance companies.

The patient’s age should be below 50, must be a local, have a minimum annual income of US$5,000, a clear medical history (should meet certain condition), and must have zero criminal records. The eligibility is then fed to a web portal along with other required information where an insurance agent takes the subscription forward if it passes the eligibility check.

Figure 9: Requirement of eligibility check and message transformation

The logical operators and if else conditional checks along with String operators can be used to evaluate the eligibility and then pass the relevant information to the web portal. The following sample illustrates the graphical representation of the above eligibility check and the transformation using the data mapper.

Sample input in XML format:

<Request>
    <name>test</name>
    <age>0</age>
    <nationality>test</nationality>
    <id>1000000</id>
    <grossIncome>100.0</grossIncome>
    <medicalConditions>
   	 <condition>
   		 <dignosis>test</dignosis>
   		 <medication longTerm="true">test</medication>
   	 </condition>
   	 <condition>
   		 <dignosis>test</dignosis>
   		 <medication longTerm="true">test</medication>
   	 </condition>
    </medicalConditions>
    <criminalRecords exits="true">
   	 <record>
   		 <penalty>test</penalty>
   		 <dateCharged>0.0.0.0</dateCharged>
   	 </record>
   	 <record>
   		 <penalty>test</penalty>
   		 <dateCharged>0.0.0.0</dateCharged>
   	 </record>
    </criminalRecords>
</Request>

In above sample, the criminal records can either exist or not exist; depending on the availability the records have to be either null or not.

Sample output in XML format:

<Request>
    	<eligability>true</eligability>
    <name>test</name>
    <age>0</age>
    <nationality>test</nationality>
    <id>1000000</id>
    <grossIncome>100.0</grossIncome>
</Request>

Figure 10: Input and output mapping of data mapper to generate the desired output

In Figure 10, comparison, logical, and string operators have been used to achieve the target of finding the eligibility of the client; the criminal records can either be present or not since they have been configured as nullable.

Use case 4: Message transformation with custom functions

Metro Sports Complex registration process for a new comer requires the name, date of birth, address, height, current weight, and current BMI. At present, when using the registration legacy system, the BMI is calculated manually. In order to provide a better service to clients, they are integrating with an online web application where the clients can register themselves and get relevant enrolment plans. The online web application is maintained by a third party and it only provides the name, date of birth, address, height, and current weight to the legacy system.

Figure 11: Requirement of the system to include BMI value to the payload

In order to solve this problem, the ESB can be used to provide message transformation. For the transformation part you can use the data mapper mediator, which provides the capability to write a custom function powered by Javascript using the custom function operator. By using this custom function, you can calculate the BMI value and send it along with the payload as required.

Figure 12: Input and output mapping of data mapper to generate the desired output

Sample input in XML format:

<request>
<name>test</name>
<dateOfBirth>0.0.0.0</dateOfBirth>
<address>test</address>
<height>1.0</height>
<weight>1.0</weight>
</request>

Sample output in JSON format:

<request>
<name>test</name>
<dateOfBirth>0.0.0.0</dateOfBirth>
<address>test</address>
<height>1.0</height>
<weight>1.0</weight>
<bmi>1.0</bmi>
</request>

The message can be transformed as required so that the back-end legacy system will receive the required information to process the registration.

Figure 13: Custom function operator that calculates the BMI

Conclusion

WSO2 data mapper is a comprehensive tool that can be used to enhance message transformation that exceeds paradigms of traditional integration logic. It also provides numerous functions that enable the message to be transformed without too much coding effort. The use case samples discussed in this article will significantly help users to discover this one-stop mediator for most message transformation purposes.