Implementing MDM Patterns on the WSO2 SOA Platform
By Asanka Abeysinghe
- 6 Jul, 2009
Master data is important for any business. Managing master data is critical as businesses makes decisions based on them. It is still valid for systems developed following the service oriented architecture styles as well. This article describes how to map MDM with SOA deployments with details of how to architecture. The WSO2 SOA platform is shipped with features required for such an implementation.
Table of Content
Wonder what master data is? Well, there are many definitions. I'd like to describe master data as reference data, excluding transactions, unstructured data, hierarchical data and meta data belonging to a business environment. Master data can be defined as nouns of a business system e.g. Employee Master, Customer Master, Location Master, Department Master. Master data is referred when creating runtime data such as transaction data. As a business grows, so does master data..
I've mentioned it before, but I'm saying it again here. Managing master data is critical for continuation. Similar to reference data, master data is also linked to all other data sets within an organization. Most organizational decisions are based on transactional data, that are in turn dependent on the accuracy of master data.
To create, manage and implement master data is not an issue in SOA, as the concepts and the principles are well supported. Issues we face with MDM and SOA are significantly different to each another. Business organizations have legacy applications and databases that stores and manage master data, created well before late 1990s when service oriented architecture style was introduced. However, today's businesses want to build SOA applications that at times still consume existing master data in the legacy applications and databases. Organizations that read these older type master data repositories, also need to update them with new values.
The traditional approach to the above problem is to use an additional database to serve new applications. This database will be synchronized with the legacy applications using a batch processers run as a part of BOD and EOD processes. The approach brings up issues related to data redundancy, where two applications may be working with two different master data sets, that are not shareable between synchronization cycles. As a result, decision making takes place with one set of data set and transactions happening on another. A solution may be for data entry users to update and verify both systems, but it is a deep time consuming activity and does not scale.
Figure 1 - Traditional Approach with batch updates
The common SOA approach is to expose master data (legacy operations and legacy data) using a wrapper service. Application developers use code generators to create services out of legacy applications and programming languages such as COBOL, RPG, Clipper etc. Code generators map the application operations to services and do the necessary data type mappings as well. However, due to poor service implementations by code generators, application developers are needed to write service proxies or plug ESB applications in front of these service wrappers. These service wrappers are called adaptors in the SOA infrastructure.
Having a common schema for shared data among the two applications is very important. WS-Transfer is a common approach to create a common schema for such systems.
Figure 2 - SOA Approach with adaptors
There are a few approaches to implementing MDM patterns in the WSO2 SOA framework. Three such implementation patterns are described below:
Figure 3 - Service Adaptors with WSO2 Stack
In this very first approach, service adaptors access the legacy application using the API provided by the application. First task of the adaptor is to transfer data access from the legacy application to a common schema. Then, services can be exposed with relevant operations in order to access the actual data. With the WSO2 stack, the adaptors can be hosted inside WSO2 WSAS, providing a rich Web services hosting and management environment. The developer of an adaptor developer can take several approaches to develop and deploy an adaptor, such as POJO, JAX-WS.
Calling the low-level API of the legacy system can be achieved using code generators possible with the tooling support of the legacy system. If the legacy API supports Java, then the adaptor could do a direct Java call. JNI will be another option to bridge to the legacy API. The most common method is to use an open standard ORB like CORBA, and do RPC calls between the adaptor and the legacy system.
Additional metadata required for the service adaptor can be stored in the repository, that is the product call WSO2 Governance Registry comes with an inbuilt repository/registry. Communication between the adaptor and the downstream SOA applications can be done using P2P, but to avoid P2P and bring flexibility on the communication layer it is recommend to use an ESB. Fast lightweight and versatile service bus available in the market comes under the WSO2 SOA framework call WSO2 ESB. By plug-in the WSO2 ESB into the solution mediation, transformation, filtering and any other logic required for the Master Data can be applied. Another advantage of plug-in the WSO2 ESB is downstream applications can use multiple transports such as HTTP/s, JMS, FTP, VFS to access the master data.
Figure 4 - Data Services with WSO2 Stack
The second pattern is a powerful data export method as services but easy to implement with the WSO2 SOA framework. Using WSO2 Data Services any database that contains a JDBC driver can expose as services. Data sets will be define using the DS query language smiler to standard SQL. User can use the same table column names with the schema or give an alias for each column. User can use a query as a operation and combine set of operations and create the service.
Similar to the above approach WSO2 ESB will be using to communicate between the Data Services and the downstream SOA applications. Configuration data required for the application will be stored in the WSO2 G-Reg by configuring it as a shared/central repository.
WSO2 Data Services ship as a functional component with WSO2 WSAS and using the WSO2 Carbon data management component application developer can configure the Master Data repository (Database). WSO2 Data Services component provides a GUI based wizard to create the data service by applying the relevant queries.
Figure 5 - Eventing with Service Adaptors
This patterns is an enhanced pattern of the service adaptors. In this, master data is sent to downstream applications using asynchronous pub/sub method using eventing. In this approach, the WSO2 ESB acts as the event-broker that contains the event-source and subscription manager. Service adaptors that publish data from the master data repository acts as event publishers. Downstream SOA applications will act as the event subscribers and event sink(s) to capture events. WSO2 G-Reg can use as the common repository/registry to store event topics, event can be publish and filter based on the topics.
WS-Eventing support in the WSO2 ESB and the ability in the the Governance Registry repository to store hiarachical resources, make this model easy to implement and efficient use. Topic-based filtering allows downstream applications to only listen for interesting events and avoid the processing of additional events.
This pattern will enhance loose-coupling of the application environment, and allows add/remove downstream applications and adaptors to the environment, without providing minimal impact to running applications.
The WSO2 SOA platform provides several patterns to implement MDM patterns in your SOA infrastructure. Having multiple patterns allow one to pick the best implementation method based on business domain, existing application infrastructure, and future enhancements of application infrastructure. Implementing MDM pattens on top of existing legacy applications and legacy data reduces data and application redundancy.
Apart from technical benifits, MDM implementations allow organizations gain huge financial benifits due to reusability of data and systems. Accuracy of decisions made and transactions performed will have huge impact on company accounts. As a result, MDM allows businesses continue work on familiar and time-tested data sets - without compromising data integrity.1
WS-* support in the WSO2 SOA platform, and the mediation capabilities of the WSO2 ESB provides additional leverage and brings value additions to the standard MDM patterns implementation.
WS-Transfer - https://www.w3.org/Submission/WS-Transfer/
WS-Eventing - https://www.w3.org/Submission/WS-Eventing/
Master data Management -  https://en.wikipedia.org/wiki/Master_Data_Management
Master Data Management -  https://msdn.microsoft.com/en-us/library/bb190163.aspx
Asanka Abeysinghe is an Architect at WSO2. asankaa AT wso2 DOT com