Category Archives: Architecture

Pull/Push Data From Central Datacenter to Reduce Deployment Complexity

The Pattern we describe in this post will be useful to organizations that operate with a central master datacenter together with distributed applications in geographically diverse locations. We can take as examples the retail sector where an organization runs a chain of many stores, hospitality sector with many hotels and restaurants, or the healthcare sector with many hospitals and pharmacies.

Synchronizing data between the distributed agent repositories and the central master repository is a common requirement for businesses of this nature. Often the two-way synchronization must occur on at least a daily basis to keep all the systems up-to-date. Transaction data has to come from the agent data stores to the master store; master data and reconciled data has to go back to the agent data stores from the master data store.

A common approach to implement the above scenario is to run a periodic process within each agent data store, which connects with a process running in the central data store and creates a channel to transfer between data stores. This approach requires a large-scale system change, requiring systematic changes to both the central system and each agent store system. It may require coordination between agent stores to avoid overwhelming the central store.  Each store may even have to purchase new hardware and the associated costs can quickly scale upward depending on the number of connected stores. IT staff may be needed to look after each and every store to keep the system running smoothly. Costs and expertise requirements mount quickly with this architecture.

How can we reduce the deployment and management complexity and keep costs reasonable?  WSO2 has identified a solution pattern that pulls and pushes data from the main datacenter, without installing additional components and initiating periodic processes in the agent data stores.

Diagram 1

This pattern consists of a central process that schedules and connects to each agent data stores using a data connectivity technology (e.g. JDBC, ADO) and directly synchronizes the data (e.g. RDBMS) running in the store. The WSO2 middleware platform, specifically the WSO2 ESB and the WSO2 Data Services Server, provides OOTB functionality to implement the above pattern.  These products are deployed a the central datacenter location.

Diagram 2

WSO2 ESB scheduled tasks are configured to kick off the synchronization task, based on the frequency of the synchronization needed. These timer tasks invoke data services deployed in the WSO2 Data Services Server, which provide a CRUD (Create/Read/Update/Delete) service interface directly against the agent data repository. The WSO2 Data Services Server is capable of executing the SQL queries or calling a stored procedure of the agent data store as required to implement the required CRUD operations. WSO2 ESB will update the sub-system data store with data coming through the data services and will push the data back through to the master store through the same data services by reading from sub-systems. Built-in mediation features of the WSO2 ESB such as transformation and routing can be used to convert messages between different data models as well as route to relevant sub-systems, kick off additional events or processes, and so forth.

This pattern is suitable for both NRT (Near Real Time) and batch synchronization requirements, whichever is best suited for the organization that runs these type of distributed deployments.

Asanka Abeysinghe, Director of Solutions Architecture
Asanka’s blog: http://asanka.abeysinghe.org/

Bus architecture to handle inbound and outbound calls with BPEL

Business processes play a major role in complex, long-running business processes in the modern enterprise. Such business processes might automate such business tasks as ordering and billing, customer or employee account provisioning, financial recordkeeping, auditing, and archiving, supply chain management, and many more.

Within SOA-based solutions a common technology for describing and executing business processes is BPEL (Business Process Execution Language). In an SOA environment, a business interaction with a user or a system results in a call to a Web service representing the business process. Such services may be implemented conveniently using BPEL deployed inside a BPEL engine such as the WSO2 Business Process Server (BPS.)

Since the BPEL engine exposes the process as a service, consumers can invoke the business process using the service interface and whichever of the bindings is most convenient. However, this integration pattern creates a point-to-point connection between the business process and the consumer – something that over time can result in “SOA spaghetti” and make management and evolution of the SOA platform difficult.

The pattern proposed here as a solution avoids this point-to-point connection by introducing a mediation layer using a bus architecture. An Enterprise Service Bus (ESB) presents a face to the consumers, takes the requests to execute the business processes and routes them to the business process services exposed by the BPEL engine. Changes to the system (either the consumers or the BPEL services) can be managed largely within the bus, simplifying versioning, new or alternate protocol deployment, monitoring, security configurations, logging and auditing, migration or scaling of services, etc. The result is more flexibility, greater robustness, and greater insight and manageability of the SOA.

Invoking external services becomes an essential part of BPEL logic. As a result, BPEL engines such as the WSO2 BPS need to connect to various other service endpoints within the service platform.

Commonly, BPEL activities are wired to service endpoints using direct partner links and service endpoint URLs. As a result, point-to-point integration is created between the business process layer and back-end services. These tightly-coupled P2P connections lead to complexity and limit system changes and enhancements, just the problem we were avoiding by fronting the BPEL service with an ESB!

To address this lack of loose coupling for our WSO2 BPS users, we’ve often used a pattern derived from the bus architecture. In a nutshell, this pattern introduces another (conceptually at least) instance of the WSO2 Enterprise Service Bus to mediate between the business processes and the back-end services.

Such a layered architecture looks like this:

Each BPS call that goes to the services layer, does so through the ESB. The ESB invokes the actual services, allowing it to manage all the endpoints and ensures traffic participates in the benefits of routing through the ESB. The diagram above shows two ESB layers. But, in a physical deployment, in most cases, it is deployed as a single ESB instance. Converting the above architecture to a bus architecture helps understand it better. Therefore, lets look at the same thing in a bus architecture.

The above diagram shows that all the upstream consumer channels, business process server (BPEL engine) and the services connect to the same ESB. The ESB wires each component.
This pattern provides a flexible and clean architecture to integrate consumer channels, business processes and backend services.

There are however a few drawbacks to this pattern which need to be balanced with the advantages discussed above. First, this will add a new component to the deployment architecture (the ESB) to be managed in the production environments. Second, two additional layers adding to the communication flows by introducing the ESB may add some latency (typically minimal) to a process invocation. Consider the consequences of these drawbacks when designing your architecture around this pattern.

Asanka Abeysinghe, Director of Solutions Architecture
Asanka’s blog: http://asanka.abeysinghe.org/

Putting SOA in a Practical Context

I was looking at some of our download statistics and was pleasantly surprised to see that one of my favorite whitepapers, Practical SOA for the Solution Architect, is also on its way to becoming our most downloaded paper.  As of this writing it still trails our eBay case study, which is another of my favorites, but Practical SOA is quickly working its way up the leaderboard.

I mentioned this to Ganesh Prasad, the author of the paper, who has a long and broad experience as an architect.  Why is this paper so popular?  Here’s what Ganesh said:

“I think the paper has struck a chord because it addresses a long-felt need. Solution Architects have been bombarded with information about SOA for years, but a lot of it is theoretical and not something they can readily put to work. The SOA education provided by vendors is more practical, but it often turns out to be self-serving because it’s meant to sell products and not necessarily to make practitioners more effective at SOA. That’s borne out by the number of ESB-based solutions out there that are tightly coupled. So again, Solution Architects are left feeling a bit short-changed because they aren’t being given a practical methodology for SOA that they can apply to their work.

“Perhaps this paper, authored by a fellow architect who has been in the same situation, addresses SOA from exactly the angle that Solution Architects want to see it addressed. They finally have an answer to their longstanding demand, ‘Give me a dead simple and practical method of applying SOA principles to my solution designs.’”

I think that’s true.  SOA has gone through a pretty wild hype cycle in the past, and the straightforward application of loose-coupling principles has sometimes suffered.  But all through that hype cycle SOA was quietly proving itself as an effective approach when thoughtfully and consistently applied.

As shiny new targets emerge for industry hype-meisters (cough – CLOUD – cough) now is a good time to organize and filter the useful legacy of SOA into a practical, cohesive, methodology.  Now is a good time to recognize that SOA has established its place in modern enterprise solutions.  If you haven’t read the paper yet, I’d encourage you to download it now!

Read more of Ganesh’s thoughts and advice at his blog: The Wisdom of Ganesh.

Disclosure: WSO2 has engaged Ganesh to create educational materials and participate in events around the topics of SOA, enterprise integration, and applying WSO2 technologies.  Given the favorable reception of this first whitepaper, I’m sure many will be waiting to see more!

Jonathan Marsh, VP Business Development and Product Design
Jonathan’s blog: http://jonathanmarsh.net/blog

Dual channeling for efficient large file processing

Recently I have come to appreciate that a pattern I’ll call “dual channeling” is emerging as a way to address a wide set of scenarios involving large files and workflows with file processing. The Dual Channeling pattern is a variation of the well known enterprise integration pattern “Claim Check”. Recently we helped a customer architect and implement a Dual Channel solution.

Businesses in domains like media/digital media, telco, printing and financial services often require large documents/files to be processed to complete a specific business function. The large file is passed through a series of steps (a workflow). The workflow adjusts to specific document types, clients or jobs. Moving the file in entirety through the workflow steps (which can be many) generally proves to be an inefficient way to manage the workflow. It creates a lot of traffic in the network and increases the time it takes to complete the workflow. Such a process typically looks like this:

sa-blog-6-img-1

The Dual Channel solution avoids this constant shipping of data by introducing two channels, one to carry the actual file and another one to carry the metadata about the file. Many steps in the workflow can then take advantage of a light-weight message with the file metadata to make the decisions and route the workflow. Workflow activities/steps can still call processes that require file processing but in this case, instead of passing the actual file, messages can pass (as part of the metadata) a reference/pointer of the file to the process. A dual channel solution might be represented like this:

sa-blog-6-img-2

To start off the dual-channel pattern, file pre-processing extracts appropriate metadata and ensures clear file identification.

Of course, the Dual Channel pattern can be implemented with the WSO2 Enterprise Service Bus (ESB). The WSO2 ESB acts as a File Transfer Gateway and a Metadata Exchange in this scenario. WSO2 Business Process Server (BPS) can be used to implement the workflows using WS-BPEL. BPEL creation by process designers is simplified with the graphical editor supported by WSO2 Carbon Studio.

sa-blog6-image3

Business process might need to execute rules to fulfill the workflow activities – and in this case the WSO2 Business Rules Server (BRS) is an ideal solution – either as a separate instance or as a feature inside either the WSO2 ESB (where rules are applied to the metadata channel) or WSO2 BPS (where the rules are part of the workflow). Enterprise deployment requirements, high-availability and scalability can be achieved by deploying the WSO2 products in cluster mode using WSO2 Carbon Clustering.

With this pattern, large and complex file processing is more efficient and rapid than ever. As the scope and scale of data explodes in the enterprise, I’m sure more and more enterprise architects will give this pattern a prominent place in their architecture toolbox. I hope it proves useful in yours.

Asanka Abeysinghe, Director of Solutions Architecture
Asanka’s blog: http://asanka.abeysinghe.org/