Dual channeling for efficient large file processing

Recently I have come to appreciate that a pattern I’ll call “dual channeling” is emerging as a way to address a wide set of scenarios involving large files and workflows with file processing. The Dual Channeling pattern is a variation of the well known enterprise integration pattern “Claim Check”. Recently we helped a customer architect and implement a Dual Channel solution.

Businesses in domains like media/digital media, telco, printing and financial services often require large documents/files to be processed to complete a specific business function. The large file is passed through a series of steps (a workflow). The workflow adjusts to specific document types, clients or jobs. Moving the file in entirety through the workflow steps (which can be many) generally proves to be an inefficient way to manage the workflow. It creates a lot of traffic in the network and increases the time it takes to complete the workflow. Such a process typically looks like this:

sa-blog-6-img-1

The Dual Channel solution avoids this constant shipping of data by introducing two channels, one to carry the actual file and another one to carry the metadata about the file. Many steps in the workflow can then take advantage of a light-weight message with the file metadata to make the decisions and route the workflow. Workflow activities/steps can still call processes that require file processing but in this case, instead of passing the actual file, messages can pass (as part of the metadata) a reference/pointer of the file to the process. A dual channel solution might be represented like this:

sa-blog-6-img-2

To start off the dual-channel pattern, file pre-processing extracts appropriate metadata and ensures clear file identification.

Of course, the Dual Channel pattern can be implemented with the WSO2 Enterprise Service Bus (ESB). The WSO2 ESB acts as a File Transfer Gateway and a Metadata Exchange in this scenario. WSO2 Business Process Server (BPS) can be used to implement the workflows using WS-BPEL. BPEL creation by process designers is simplified with the graphical editor supported by WSO2 Carbon Studio.

sa-blog6-image3

Business process might need to execute rules to fulfill the workflow activities – and in this case the WSO2 Business Rules Server (BRS) is an ideal solution – either as a separate instance or as a feature inside either the WSO2 ESB (where rules are applied to the metadata channel) or WSO2 BPS (where the rules are part of the workflow). Enterprise deployment requirements, high-availability and scalability can be achieved by deploying the WSO2 products in cluster mode using WSO2 Carbon Clustering.

With this pattern, large and complex file processing is more efficient and rapid than ever. As the scope and scale of data explodes in the enterprise, I’m sure more and more enterprise architects will give this pattern a prominent place in their architecture toolbox. I hope it proves useful in yours.

Asanka Abeysinghe, Director of Solutions Architecture
Asanka’s blog: http://asanka.abeysinghe.org/

What an awesome day! The grand opening of WSO2 Palo Alto.

IMG_0527The WSO2 team is thrilled to announce the opening of our new Palo Alto California office!  To accommodate our increasing industry and Silicon Valley presence, we’ve expanded our California office location and personnel, and today marked our move from Mountain View and the first day in our new office location!

We’ll drive more of our North American marketing from this location, as well as expand our sales and support reach in this time zone.

Palo Alto is right in the epicenter of innovation and entrepreneurship in Silicon Valley, appropriate to our goal of reinventing middleware and the way businesses leverage information to succeed in the challenging global marketplace.  And, with this location we have a great environment to work from – including being directly above a Starbucks!

IMG_1130A whole WSO2 crew was on hand to help us celebrate, from all four WSO2 continents:

Back row: Devaka (Sri Lanka), Sanjiva (Sri Lanka), Paul (UK), Chamith (Sri Lanka), Hasmin (Sri Lanka), Samisa (Sri Lanka); Middle row: Monica and daughter (California), Udeshika (Sri Lanka), Puny (Sri Lanka), Kushlani (Sri Lanka); Front Row: Jonathan (California), Mahesh (Australia)

IMG_0516-1After a day of final organization and moving into the space, we held a small office opening ceremony, including taking turns lighting a traditional coconut oil lamp (Samisa demonstrates) followed by sharing some Sri Lankan delicacies – I still don’t know what magic the WSO2 ladies used to conjure them up.  And cake!

IMG_0521This opening marks another significant milestone in the ongoing success of WSO2 and would not be possible without the diligent effort of the whole global team of engineers, sales, marketing and operational staff, and of course the best family of customers in the world!  Today we salute you with a piece of cake and a Starbucks, and hope you will join us in celebration from whatever continent you are on!

Jonathan Marsh, VP Business Development and Marketing
Jonathan’s blog: http://jonathanmarsh.net/blog

Get Ready for Summer – SOA Style

As you gear up for summer, sign up for our ever-popular SOA Summer School program starting in June.

No idea what SOA Summer School is? Well it’s a two-month program that offers weekly online sessions on various topics relating to enterprise SOA. We started WSO2 SOA Summer School in 2009 to help IT architects and developers beat the recession and imageupdate their knowledge and insights into the latest SOA technologies and best practices. We got such great feedback, we ran it again last year and are expanding it with brand new content this year.

While we’ve covered basic technology in the previous sessions, this year we focus on practical solutions to real world challenges faced by enterprises today. We’ll showcase WSO2-based solutions, from security to governance to enterprise integration and cloud. We’ve also changed the format a little bit as well — the sessions are now more intensive and only two hours long.

Here’s the list of sessions that you can sign up for by going to http://wso2.org/training:

  • Security policy enforcement for the enterprise
  • Identity Management in the Cloud
  • End-to-end governance in the enterprise
  • Enterprise integration with SAP and WSO2 ESB
  • Enterprise Integration with the FIX Protocol
  • Mobile-izing enterprises with the WSO2 Mobile Services Gateway
  • Master Data Management in your SOA
  • Platform-as-a-Service: The WSO2 way
  • Wrap-up: The Best of Summer School 2011

Here the link to the press release we issued this week, for more info.

Hasmin AbdulCader, Director, Marketing
http://www.twitter.com/hasmina

Open Conversations on the New Business of Enterprise Software

WSO2