Introducing the New Visual Data Mapper for WSO2 Micro Integrator

Introduction

We’re excited to introduce a significant addition to WSO2 Micro Integrator: our new visual data mapper, now available with the 4.3.0 release. This feature is designed to enhance your data integration experience by providing a more intuitive and user-friendly way to manage and transform data.

Why a Visual Data Mapper?

Data mapping and transformation can often be complex and time consuming. Traditional methods usually require manual coding or involve intricate configuration files to specify mapping logic. Our new visual data mapper simplifies this process by offering a graphical interface that allows you to see and manage your data flows more easily. This not only speeds up development but also reduces the potential for errors.

Key Features

This new data mapper is accessible with the Micro Integrator VSCode extension (Visual Studio Marketplace - Micro Integrator), as the Data Mapper mediator. When compared to the eclipse-based old data mapper available in previous versions, this new tool offers significant features and improvements, enhancing the integration flow.

• Code is the Diagram

The data mapper leverages TypeScript to ensure that the code itself serves as the diagram. With TypeScript code as the single source of truth, there are no intermediate representations needed. Any manual code edits are instantly reflected in the visual mapping, providing a seamless experience between coding and diagramming. This approach ensures that your mappings are always synchronized with the underlying code. Moreover, this allows AI tools such as CoPilot to automatically generate and refine mappings, enhancing the efficiency and accuracy of the process.




Figure 1: Side by Side view of Diagram and Code

• Type Safety

By using TypeScript underneath, the data mapper guarantees type safety for mappings. TypeScript’s strong typing and diagnostic capabilities swiftly identify type incompatibilities and errors, providing immediate feedback. This proactive error detection helps you address issues early, significantly enhancing the reliability and accuracy of your data transformations.




Figure 2: Type Incompatible Diagnostics

• Scalable UI Design

Transforming data frequently involves handling nested fields and arrays, which can be particularly challenging when dealing with lengthy input and output payloads. The new data mapper simplifies this complexity with its streamlined mapping design where you can navigate into your array-to-array mappings and create mappings in focused views. Regardless of how complex the arrays are, whether multi-dimensional or containing intricate objects, the new design scales effectively to manage these complexities. Furthermore, you can effortlessly filter input and output fields, and independently scroll through them, making it easier to locate and map the necessary elements.




Figure 3: Navigating Array-to-Array Focused Views

• Reusable Sub Mappings

Handling repetitive mapping logic across multiple output fields can be time-consuming and error-prone. To address this, our data mapper introduces reusable sub-mappings. This feature allows you to create and reuse common mapping logic across different fields, saving you time and effort while ensuring consistency and efficiency in your workflows.




Figure 4: Using Sub Mappings

• Expression Editor within Mapping UI

Often, most data mapping tools require opening a form or mapping file to add complex logic, which can be cumbersome and disrupt developer experience. Our expression bar overcomes this by providing contextual code completions, enabling you to write and refine transformation logic directly within the interface. This feature quickly helps you find the necessary functions and operators, reducing the learning curve and significantly enhancing productivity.




Figure 5: Expression Editing Within Mapping UI

• Array Filtering Based on Conditions

Easily filter arrays by specifying conditions to extract only the elements you need. This feature simplifies data transformation by allowing you to apply precise filters directly within the mapping interface, ensuring that your output includes only the relevant data.




Figure 6: Filter Input Array Before Exact Mapping

• Predefined Operators for Commonly Used Transformations

The data mapper comes with a set of predefined operators that cater to common data transformation needs. These operators make it easy for you to quickly apply standard transformations without the need to write extra code, simplifying your workflow and saving you time.




Figure 7: Using Predefined Operators

• Ability to Use Custom Functions

Users can define and utilize custom functions within the data mapper. This flexibility allows for the creation of tailored transformations that meet specific business needs.




Figure 8: Using Custom Function

Now let's try out the new Data Mapper in a Micro Integrator project:

1. Open a Micro Integrator project on Visual Studio Code Editor. Add an API with a POST resource to receive payloads for the Data Mapper.



Figure 9: Create an API Endpoint

2. Click on the plus button at the API flow and add the Data Mapper mediator.



Figure 10: Create a New Data Mapper

3. Now provide input and output samples to define input and output types for mapping data.



Figure 11: Configure Input and Output

4. Afterwards, you can define input and output types and start mapping.

How to Create Mappings

There are three types of mappings you will do with the visual data mapper:

One to One Mapping

An input field is directly mapped to an output field. Simply click on the input field and then select the output field to create the mapping.

Diagnostics are indicated for type mismatched mappings and users can fix it by using the expression bar.

As an example, figure 12 shows how to do a simple direct mapping to map a number-typed input field to a string output field.

Figure 12: One-to-One Mapping

Many-to-One Mapping

The output field is generated by using two or more input fields. This kind of mapping can be mostly seen in cases where the output field is derived from a concatenation, conditional expression, function call with multiple arguments, etc.

A simple string concatenation can be done as depicted in the figure below.

Figure 13: Many-to-One Mapping

Array to Array Mapping

An array-typed input field is mapped to another array-typed output field. This can start similarly to one-to-one mapping by simply creating a direct map between the input and output fields. Then by using the ‘map with array function’ option users can go into a focused view to further map array elements.

Figure 14 shows how to create an array-to-array mapping between two different types of input and output arrays.

Figure 14: Array-to-Array Mapping

Conclusion

Data Mapper provides many more features to facilitate comprehensive data mapping scenarios. The WSO2 MI documentation has everything you need to explore and utilize the visual data mapper fully.

Our roadmap includes exciting new features such as real-time data preview, AI-based data mapping, on-the-go outputs without pre-defining the expected output, and many more. Stay tuned for updates and get involved in our community forum to contribute your ideas and feedback.