Dual channeling for efficient large file processing

Recently I have come to appreciate that a pattern I’ll call “dual channeling” is emerging as a way to address a wide set of scenarios involving large files and workflows with file processing. The Dual Channeling pattern is a variation of the well known enterprise integration pattern “Claim Check”. Recently we helped a customer architect and implement a Dual Channel solution.

Businesses in domains like media/digital media, telco, printing and financial services often require large documents/files to be processed to complete a specific business function. The large file is passed through a series of steps (a workflow). The workflow adjusts to specific document types, clients or jobs. Moving the file in entirety through the workflow steps (which can be many) generally proves to be an inefficient way to manage the workflow. It creates a lot of traffic in the network and increases the time it takes to complete the workflow. Such a process typically looks like this:

sa-blog-6-img-1

The Dual Channel solution avoids this constant shipping of data by introducing two channels, one to carry the actual file and another one to carry the metadata about the file. Many steps in the workflow can then take advantage of a light-weight message with the file metadata to make the decisions and route the workflow. Workflow activities/steps can still call processes that require file processing but in this case, instead of passing the actual file, messages can pass (as part of the metadata) a reference/pointer of the file to the process. A dual channel solution might be represented like this:

sa-blog-6-img-2

To start off the dual-channel pattern, file pre-processing extracts appropriate metadata and ensures clear file identification.

Of course, the Dual Channel pattern can be implemented with the WSO2 Enterprise Service Bus (ESB). The WSO2 ESB acts as a File Transfer Gateway and a Metadata Exchange in this scenario. WSO2 Business Process Server (BPS) can be used to implement the workflows using WS-BPEL. BPEL creation by process designers is simplified with the graphical editor supported by WSO2 Carbon Studio.

sa-blog6-image3

Business process might need to execute rules to fulfill the workflow activities – and in this case the WSO2 Business Rules Server (BRS) is an ideal solution – either as a separate instance or as a feature inside either the WSO2 ESB (where rules are applied to the metadata channel) or WSO2 BPS (where the rules are part of the workflow). Enterprise deployment requirements, high-availability and scalability can be achieved by deploying the WSO2 products in cluster mode using WSO2 Carbon Clustering.

With this pattern, large and complex file processing is more efficient and rapid than ever. As the scope and scale of data explodes in the enterprise, I’m sure more and more enterprise architects will give this pattern a prominent place in their architecture toolbox. I hope it proves useful in yours.

Asanka Abeysinghe, Director of Solutions Architecture
Asanka’s blog: http://asanka.abeysinghe.org/

What an awesome day! The grand opening of WSO2 Palo Alto.

IMG_0527The WSO2 team is thrilled to announce the opening of our new Palo Alto California office!  To accommodate our increasing industry and Silicon Valley presence, we’ve expanded our California office location and personnel, and today marked our move from Mountain View and the first day in our new office location!

We’ll drive more of our North American marketing from this location, as well as expand our sales and support reach in this time zone.

Palo Alto is right in the epicenter of innovation and entrepreneurship in Silicon Valley, appropriate to our goal of reinventing middleware and the way businesses leverage information to succeed in the challenging global marketplace.  And, with this location we have a great environment to work from – including being directly above a Starbucks!

IMG_1130A whole WSO2 crew was on hand to help us celebrate, from all four WSO2 continents:

Back row: Devaka (Sri Lanka), Sanjiva (Sri Lanka), Paul (UK), Chamith (Sri Lanka), Hasmin (Sri Lanka), Samisa (Sri Lanka); Middle row: Monica and daughter (California), Udeshika (Sri Lanka), Puny (Sri Lanka), Kushlani (Sri Lanka); Front Row: Jonathan (California), Mahesh (Australia)

IMG_0516-1After a day of final organization and moving into the space, we held a small office opening ceremony, including taking turns lighting a traditional coconut oil lamp (Samisa demonstrates) followed by sharing some Sri Lankan delicacies – I still don’t know what magic the WSO2 ladies used to conjure them up.  And cake!

IMG_0521This opening marks another significant milestone in the ongoing success of WSO2 and would not be possible without the diligent effort of the whole global team of engineers, sales, marketing and operational staff, and of course the best family of customers in the world!  Today we salute you with a piece of cake and a Starbucks, and hope you will join us in celebration from whatever continent you are on!

Jonathan Marsh, VP Business Development and Marketing
Jonathan’s blog: http://jonathanmarsh.net/blog