Introducing the Visual Data Mapper

We are pleased to announce that we have introduced a Visual Data Mapper for Choreo. This is available through the Ballerina extension for Visual Studio Code. You can use it to convert data between input type(s) and an output type.

Data mapping lets you translate data from one form to another, such as from one JSON format to another. The data mapping process is crucial to enterprises as it’s needed for the following areas: 

Data integration: This involves moving data from multiple sources to a centralized location (done at regular intervals) while removing redundant data fields and duplicate entries. The goal is to create a comprehensive data set that can be shared with users across the organization.

Data migration: This process is where we move data from one location to another as a one-time task with the goal of retiring the existing storage system and using the new location permanently after migration. For example, this occurs when an organization moves from an on-premise storage to a cloud provider.

Data transformation: This involves translating data from one format to another while cleansing, restructuring the data, and performing transformations such as aggregation, enrichment, etc. This is usually done to enable communication between different systems that expect different forms of the same data or to collect data in an appropriate format for analysis.

There are many types of visual data mapping tools available, and they often use template or configuration files to persist the logic. These files are fed into a template engine that handles the transformations. However, when it comes to complex conversions, users need to write custom functions and refer to them inside the template file. These files can be very complex and aren’t meant to be edited manually. Furthermore, custom functions may also need to be written as well. 

On the other hand, any programming language can be used to implement a data mapper as a function. It can then be shared as a library or exposed as a web service. Programming languages offer more flexibility compared to template files, but they usually lack visual tools that help users with conversions without needing to touch the code. Choreo’s Visual Data Mapper offers the flexibility of a programming language plus the user friendliness of a visual tool.

Visual Data Mapping is More Powerful Than Writing Code

Visual data mapping enables you to map via an intuitive user interface and prevents the need to be aware of the programming logic underneath. The Visual Data Mapper can generate a Ballerina source code as you map via the user interface or vice versa. It allows developers to focus only on the business logic and hides the complexity of the programming language. This makes the data mapping process faster and less error prone. 

The Ballerina source code is the single source of truth for the Visual Data Mapper. Hence, it lets you open and edit existing data mappings made via the source code without changing the user experience. This is shown in Figure 1 below.

Figure 1: Split View of Code and Diagram

Furthermore, it provides a powerful expression editor, which you can use to generate the required Ballerina expressions to map data with conditional values, execute type conversions, string manipulations, date-time parsing, and much more. 

As shown by Figure 2, the Ballerina expression editor also lets you discover and call built-in Ballerina utility functions available to process values. Depending on the context, it can suggest any applicable fixes, expression templates, inputs, operators, and functions available.

Figure 2: Visual Expression Editor

Data Mapping is Everywhere

When you develop services or applications, it's common to talk to different internal and external systems or services. These systems expect or produce data in various forms and data mapping is required when you need to pass external data to these systems or when you need to expose the data from them to external parties. 

While there may be several use cases, let’s focus on the given examples below. 

• Integrating different services

When you are performing an integration between two different services, you must convert the response from one service to a different form, so the second service can be called with it. 

For example, you may wish to review potential new leads from Salesforce and append them as rows to Google Sheets. Data mapping helps to convert the collection of Salesforce leads to a two dimensional string array which is the expected data format in the append rows API of Google Sheets.

• Aggregating or segregating

Occasionally, you may need to create a new service which serves aggregated data from many other services. You may need to collect all the data from some services and parts from others. In this case, data mapping is needed to create the final output from several different inputs, by taking data from some as a whole and only a few selected fields of data from others.

• Transforming the persistence layer

Let’s say you want to develop a Create, Read, Update, and Delete (CRUD) service and you are using a relational database for persistence. The normalized database expects data to be in a non-redundant format to keep its integrity, but for the clients of the service, usually you need to serve or accept data in a different format for the best user experience. You will need data mapping to do these conversions for relational databases and every other persistence system.

When Should you use Data Mapping in Choreo?

 Here are popular situations where you can use Choreo’s Visual Data Mapper. 

• Using connectors
  

In Choreo, you can use connectors to communicate with external or internal services in your organization. These connectors define the request and response types for each of the APIs available.

When you create an integration using these connectors, once you have added the relevant connector actions, you can create a data mapper to convert the response from one connector action to a request to another.

• Using webhooks

You can use built-in webhooks in Choreo to listen to events from popular cloud services. Like connectors, these webhooks already have defined data types of each event they can listen to. Once you configure a webhook within the event handler, you can access the event payload and use a data mapper to convert that into an expected format and call another service, using the built-in connector, with that data. 

• Creating services 

Developers can create REST or GraphQL services in Choreo. When creating these services, you can use the record editor to create data types to describe input and output payloads. Once you implement each resource, you can use data mapping to convert request payloads to process the inputs or prepare response payloads from other available data sources.

Data Mappings Supported in Choreo’s Visual Data Mapper

There are various types of conversions you can do, but they can all be categorized into three common categories.

1. One-to-one mappings

A source field can be directly mapped to a target field. If the data types of the source and target mismatch, use the expression editor to convert the input value to the expected type.

For example, as shown in Figure 3 below, an integer field from input is mapped to a string field in the output with the help of expression editor suggestions. 

Figure 3: One-to-One Mapping

2. Many-to-one mappings

In this type of mapping, several input fields can be used to generate the value of an output field. The inputs can be combined using various operators available. For example, we can combine, or concatenate two string input fields to produce the output string as seen in Figure 4.

Figure 4: Many-to-One Mapping

3. Collection-to-collection mappings 

This supports mapping between arrays and other collection types. Collection-to-collection mappings can be treated as a one-to-one mapping as well. However, the target collection often needs members to be in a different form than in the source collection. The Visual Data Mapper lets you easily achieve this as shown in Figure 5.

Figure 5: Collection-to-Collection Mapping

Getting Started

Open the Ballerina package for your component using Visual Studio Code. Make sure you have installed the Ballerina Extension (Ballerina - Visual Studio Marketplace) beforehand.

Go to the low-code editor, click the plus icon in the canvas and select “Data Mapper”.

Figure 6: Opening Visual Data Mapper

As seen in Figure 7, give a name for the data mapper and select input and output types. You can create these from scratch using samples or re-use existing types from the connector or webhook package.

Figure 7: Configuring Visual Data Mapper

Please follow the Data Mapper guide for detailed information.

Find out more today by signing up to Choreo for free!