Enterprise Application Integration (EAI) has been described as “the unrestricted sharing of data and business processes throughout the networked applications or data sources in an organization”. An integration solution must be capable of coordinating different entities that are unpredictable and asynchronous in nature. From a technical point of view, such systems require the support of long running processes with persistence capabilities and should be able to correlate asynchronous request responses.
WSO2 Enterprise Integrator has an ESB profile to handle synchronous messaging, whilst the business process profile supports asynchronous system integrations and long running processes.
Business Processes in an Enterprise Environment
Enterprises automating their business processes is a common occurrence today and there are many benefits to this. A few of them include:
- Implementing complex processes - with excellent tooling and notations, it is easy to design and implement complex processes using business process concepts.
- Fast-changing processes - business processes require frequent changes due to various constraints and improvements. A BPM solution supports rapid change application, to enable an enterprise to adapt quickly to these changes for better customer satisfaction and improved productivity.
- Moving towards enterprise mobility - automated processes enable employees to work remotely, supporting better employee satisfaction and helping them become more productive at work.
- Improved customer service - well-defined business processes enhance an enterprise’s performance with the added benefit of improving customer service.
Business process management is a continuous cycle. The following diagram illustrates the key phases of business process management:
Figure 1: Key phases of BPM (Image source - searchcio.techtarget.com/)
The cycle begins with the proposed process model, which is implemented after analysis. These initial phases must be understood by business people (and not only technical ones). It is a good practice to monitor processes while they are in progress to identify any pain points and address them to improve the processes.
BPMN over BPEL
Figure 2: BPMN vs BPEL target stakeholders
BPEL is an XML based language which is highly coupled with the WSDL based SOAP web services. BPEL supports invoking web services, correlating asynchronous messaging systems, persisting process details, and the handling of exceptions recovering. A BPEL engine can be considered as a web service orchestration engine.
BPMN 1.0 was introduced as a process modeling notation for business users. This was followed by BPMN 2.0 which introduced the execution aspects of the specification. . BPMN targets both business users and the process implementers, has its own execution, not bound to any specific technology, and has other extension points to integrate with other technologies. BPMN should be your choice over BPEL whenever possible due to a number of reasons:
- BPMN is readable by both business and technical users, whereas BPEL is more complex and not easily understood by non-technical users. This is a disadvantage because it minimizes the participation of business people when it comes to process design and optimization phases.
- Developers need a good understanding of WSDL, SOAP, and XML schema concepts to learn BPEL. This is not the case with BPMN, and hence it is easier to learn.
- BPMN is easily extensible and has the freedom of extension via service tasks. As such, the implementation model can decide how to provide the required extensions. For example, we can define a REST service task using Java to invoke REST endpoints. BPEL requires the use of SOAP services to invoke external services.
- Most of the existing BPMN implementations support REST based integrations. Since WS based concepts are on the wane at present, BPEL implementations that are restricted to WS based concepts are becoming depreciated.
(Do note however that BPEL can be a good choice if you need to orchestrate legacy systems using SOAP based web services).
Using BPMN in Your Enterprise Integration Solution
Let’s take a detailed look at some of the most commonly required use cases that can be achieved using BPMN concepts and how these are modeled in WSO2 Integration tooling.
Integrating With Long Running Processes
In some instances, an integration flow cannot be completed immediately and may require time for completion - ranging from minutes to months. There could be a requirement to monitor the status of the flow, the need to rollback the flow upon a request, change the flow after a timeout, etc. Maintaining the state of the integration flow is important in these scenarios. BPMN engines persist each step of the workflow. Then the flow can be even visualized with the current state, which path it has taken, etc.
To explain this process more clearly, let me use the example of a shipping package tracking system. Stakeholders may want to know where the package is at any given time.. This can be simply modeled as the status of the BPMN process. For an instance, we can model each transitioning place of the package as a user task and assign the task to the agents in each location. They can then complete their tasks by adding the arrival and departure times.
Figure 3: Package tracking workflow with the mail notifications
Another example is an order processing system. Suppose a customer realizes that his/her credit card has expired at the point of payment. Rather than cancelling the entire order, we can maintain the current state of the order until he/she renews the credit card and continue with the order at a later time.
Message Correlation and Coordination
When dealing with asynchronous communications, there are situations where we need to correlate messages with the existing integration flow. We may have to merge several messages in the same integration flow, decide whether or not to start a new flow or integrate with an existing flow based on the message content. There could be situations where the integration flow in our scope may have been started by another parent process and we may need to coordinate our flow with the parent process as well.
I will use a series of diagrams to explain these processes.
Figure 4: Modeling a message sequence. This process starts upon receiving a message and then waits for the second message.
Figure 5: Modeling a message synchronization. This is done using a parallel gateway and the execution does not take place until both messages are received.
Figure 6: Use message exclusion if you have to wait for two different messages, while continuing the flow once one of the messages has been received.
Figure 7: We may need to wait for a message only for a defined duration. This diagram illustrates how you can do timeout handling .In this instance the timer event will conclude if the message has not been received within the defined time and the process will continue.
Figure 8: The coordination with a parent integration flow/process also can be implemented by adding a signal boundary event to terminate the process. Then the parent process can send the relevant signal to terminate the process. This figure shows you can model a process that can be terminated by sending a signal event. A catching boundary signal event has been attached to the sub process. In this scenario, a terminate end event has to be used in the signal event flow to terminate the parent process.
An integration solution only handles technical exceptions such as the connectivity issues. However, there are business errors as well. BPMN is capable of handling such errors out of the box, using the error boundary event.. The error boundary event can be attached to any task or a container, following which we can decide what action to take after an error occurs through the outgoing flow of the boundary event. We can propagate the error to the parent process or the scope, change the flow of the process, perform a retry or we can start a new process with an error start event as well.
Figure 9: If an error occurs, a user task is created to record the error and the process is terminated as an error end.
When we model a business process, it is required to handle business transactions as well. An atomic business transaction can be a single activity in BPMN or a set of activities. Let’s take the example of a travel reservation process.. The process may continue and attempt to book a flight and a hotel for a particular destination. However, no hotels may be available for the required dates. In such a scenario, the flight reservation will have to be cancelled.
BPMN provides the compensation event to handle such scenarios.
Figure 10: We can add the compensation boundary event to an activity or a sub process to add a compensation flow as shown in this figure.
BPMN Modeling Best Practices
As with any software project, a proper requirement analysis must be conducted prior to starting the process modelling. You can begin the modelling once you have answers to the why, when, what, who, and where questions of the process. A clear understanding of the actors of the process and scope you are going to model is important. If you are going to model several scopes, then you must ensure that you use pools, swimlanes, etc. to illustrated the different scopes. It is a good practice to not over model as well. For example, there is no point of modeling a manual process that is done by a single actor, as we can keep it as a single manual task.
Finish everything you start
According to the BPMN specification, start event and end event are not mandatory. However, it is better to start a process with one more start event and make sure all the flows finish with an end event. Not having end events may result in issues and misinterpretations. Having them in your flow adds more readability to the workflow.
Use a systematic naming process
BPMN is relevant for both business and technical users and as such, the clarity of a BPMN model is very important. Each BPMNmodel has its own generic meaning. When labeling them, try to use detailed explanations of the logic in the workflow.For example,, when you have several end events in the same process, naming all of them as end event does not mean anything. Specify what is happening in a task with a verb and a noun. Do not label gateways, they are not tasks - the symbol of the gateway denotes its functionality. Label the outgoing paths from gateways.
Use correct end events
When there are multiple flows in a process, there are multiple end events. The type of end event which is used is important in process execution. If you need to finish only one flow while another flow is running, use the normal End event. If you use a Termination End event, it will terminate the whole process. Hence, we must always select the correct end events.
Use sub processes effectively
When we have a repetitive set of tasks, we can model them as sub tasks and invoke the same sub process from several places. This will minimize the size of the diagram. Such sets of tasks can be done by a single section in an organization, which makes defining the operations of such operations easier.
Sub processes are used to define the process scopes. Pools and Swimlanes can be used to define participant based scopes. Even though these are not very significant in process execution, they are very useful for a clearer understanding of the process.
Maintain consistency of the default flow
The layout of a diagram is very important for the readability of the diagram. Ensure that the main flow of the diagram is consistent, avoid crossing lines, use a horizontal flow of the sequence, and add message flows vertically.
Consistent use of gateways
Use the same type of gateways to split and merge the flows. When you draw the gateways keep a balance between the split and merging gateways. The corresponding gateways (splitting and merging) should be clearly visible. This is very helpful to understand the logic of the BPMN workflow.
BPM Support in WSO2 Enterprise Integrator
Business Process profile has incorporated capabilities of WSO2 Business Process Server (BPS) into WSO2 Enterprise Integrator (EI). Business process profile can execute business processes based on the BPMN specification by OMG group or BPEL (Business Process Execution Language), which is an OASIS standard. WSO2 EI tooling supports the designing of business process diagrams with a drag and drop editor. It also has monitoring capabilities with an in-built set of statistics which are integrated with the analytics profile of WSO2 EI. For more information on modeling BPMN with WSO2 BPS, you can refer to this article: “How to Model BPMN Business Processes with WSO2 Business Process Server.”
Monitoring business processes execution is one of the main phases in BPM. It enables you to optimize your processes and achieve more productive business life cycles. By monitoring processes, you can identify reusable sub processes of the process, improvements to resource allocations, parallelizable processes, and bottlenecks.
BPMN analytics capabilities that are introduced in WSO2 EI are capable of monitoring your BPMN process execution. Apart from that, there are limited monitoring capabilities from BPMN explorer web application as well.
When running the business profile of WSO2 EI, you need to access the Reports tab in: URL https://:9445/bpmn-explorer/reports. Click on the Generate reports button to generate some predefined reports based on available process details. These reports provide details on the active process instance count, the average time taken for each process, etc. There are also process specific reports with the breakdown of the time taken for each task. These reports allow us to identify the bottlenecks of the enterprise which the management can use to support improvements.
More refined analytics information is available in the WSO2 EI analytics profile. There are two widgets available in the analytics portal - process analytics and user analytics. Business Process analytics has three sections - Process Monitoring, Task Monitoring, and User Monitoring.
Figure 11 - Business Process Analytics
Process monitoring can be used to:
- Determine the distribution of process execution times.
- Identify the frequently invoked tasks.
- Measure the workload of the users in the organization.
- Measure the results of process changes/improvements using process versions related filters.
Task monitoring can be used to:
- Measure the performance of users when performing tasks.
- Determine the variation in demand at a particular time over a selected period of time for tasks.
- Determine the distribution of the execution times of selected tasks.
User monitoring page can be used to:
- Evaluate the expertise of users performing different tasks.
- Identify the different processes in which selected users are participating.
- Identify the capability of users participating in different processes.
Please refer the documentation on Business Process Analytics for more information.
Business processes are an essential part of enterprise integration since due to the high levels of human involvement and asynchronous invocations involved in enterprise automations. When a BPM solution is required, it is important to design business processes with the involvement of all stakeholders. After modeling and executing the business process, it must be continuously monitored and existing processes must be optimized to achieve the maximum productivity in your organization. This can be achieved with WSO2 enterprise integration which supports the modeling, execution, and monitoring phases of BPM.