WSO2 ESB: Guaranteed Delivery with Message Store Message Processor

Table of contents

• Introduction to MSMP
• When to use MSMP
• How to use MSMP
• Implementation of MSMP
• Improvements and future work

Introduction to MSMP

WSO2 ESB’s Message-stores and Message-processors are used to store incoming messages and then deliver them to a particular backend with added Quality of Services (QoS), such as throttling and guaranteed delivery. In other words, this is an implementation of Store and Forward of messages. Since the entire story of MSMP is around Store and Forward of messages let’s first understand what these mean.

Store and Forward of Messages

As its name implies, it is used to store incoming messages and later forward the message to an intended recipient. The main advantage of this approach is that it allows to send messages reliably to back-end services. These messages can be stored in any reliable storage, such as databases, file system, and combination of both. In our case, the preferred approach is to use JMS queues as it support persistence of messages as well as reliable retrieval of messages. Once the messages are stored, Forwarders retrieve the stored messages at a given rate and sends it reliably to the desired backend. To ensure reliability, Forwarders encompass techniques, such as retrying of failed messages. Since you have the big picture, let’s see how WSO2 ESB has implemented this technique along with a few other capabilities.

WSO2 ESB uses three basic constructors to implement this paradigm: store-mediator, message-store. and message-processor.

Store mediator

As it name implies, it is a mediator of the synapse engine. This mediator is used to store messages in Message-stores. So, if one wants to store messages, he/she would have to use this mediator within a sequence. This could be any sequence - an in-sequence, out-sequence, or even a fault sequence.

Message store

By the looks of it, you may think this is where we store messages, but we don’t. In fact, this is an interface, in other words a proxy to the actual implementation of message-store such as in-memory map, JMS queue, etc. Apart from acting as a proxy, message stores are used to manage and provide consumers and providers that are used by store mediator and message-processors.

Message processor

It is used to consume messages from the message-store and then deliver it to a configured backend. When a message-processor sends messages to the backend, we have more control over the message sending part. For example, we can throttle the speed of the delivery of messages or we can reliably deliver messages to a configured backend.

That’s a basic understanding of the components, and let’s now see how they are orchestrated to archive the desired functionality.

Figure 1

Figure 1 shows the end-to-end message flow in the MSMP setup, starting from the client to the backend. Up to this point, you should have got a basic idea about MSMP and its components. We now take a look at real-world usages of MSMP.

When to use MSMP

The main purpose of using MSMP is for archiving guaranteed delivery, or in other words, to archive reliable messaging. Therefore, whenever you come across the requirement of guaranteed delivery in the context of WSO2 ESB this is something that should come to your mind. For instance, there could be scenarios where the availability of the backend service is uncertain, or even though the backend service is available, there could be application level failures. Therefore, in such situations, and in order to avoid message losses, one can use MSMP.

Another practical usage of MSMP is message throttling. There could be scenarios where the backend is incapable of handling high loads of messages or the provided bandwidth for the backend is limited. In these instances one can use MSMP to throttle the rate of message delivery to the backend, which in turn also ensures guaranteed delivery.

MSMP can also be used to implement certain Enterprise Integration Patterns (EIP), e.g. Dead Letter Channel (DLC) EIP. The concept of DLC EIP is quite simple; let’s say you send a message to a particular backend and if the backend is down you place the message in a queue. Afterwards, you can retry sending those messages to the backend. This way you can ensure that there won’t be any message loses. For more information about DLC EIP read [1].

How to use MSMP

In-memory message store over JMS message store

First thing to decide when using MSMP is whether to use In-memory message-store or JMS message- store. In-memory message-store is a primitive form and is not recommended for use in production environments. There are two main reasons for this - 1) since the In-memory message-store resides within the ESB, it could take up all the heap allocated to ESB as it grows with messages; 2) in case ESB goes down for some reason, all messages stored in In-memory message-store will be lost.

On the other hand, JMS message store uses JMS message brokers, such as ActiveMQ or any other JMS-compliant message brokers. These message brokers are specifically engineered to store messages for later use. A few of the main advantages of having a JMS message store are as follows:

1. You can deploy the message broker on a different node from the ESB node; therefore, even if the ESB node goes down, your messages will not be lost.
2. These brokers support message persistence, which makes sure that even if the broker node itself goes down, there won’t be any message losses.
3. These brokers have special support for reliable messaging, which is exactly what we are trying to support with MSMP. In fact, the JMS part of the MSMP is implemented by leveraging these features. Therefore, MSMP can guarantee zero-message-lose even when the message broker is running on a different node.

Given the above-mentioned pros, it is always recommended to use JMS message-store in production environments.

Sampling processor vs Forwarding processor

This is something that most users get confused with. There are a few significant differences between Sampling-processor and Forwarding-processor. To get a better understanding, firstly, let’s look at the characteristics of each message-processor.

Sampling processor

• Takes a sequence, which means the consumed message is sent through a synapse sequence, which in turn can transform the message into something else before sending it to the backend service.
• Based on non-blocking implementation, which means it has a better performance.
• Supports throttling of messages.

Forwarding processor

• Takes an endpoint, which means the message is sent to the backend as it is stored in the message store; however, it is possible to apply any transformations on the message before storing it in the message store.
• Based on blocking implementation, which means it only can send one message at a time.
• Supports throttling of messages.
• Supports reliable messaging.
• Can handle REST scenarios better.

As you can see they are different in behaviour. However, Forwarding-processor is more widely used than the Sampling-processor as it has more capabilities than Sampling-processor. So, if there is a situation where you cannot decide which to use, it is always safer to go with the Forwarding-processor.

Hello-World Example of guaranteed delivery with MSMP

This is just to give you a look and feel on what the actual synapse configurations look like for MSMP. Detailed samples of using MSMP can be found at link [2] and [3]. In this example, we will be using MSMP to store an incoming message and then forward it to a back-end service.

Step 1 - Adding Message store

Let’s start with the JMS Message-store. In this example, we’ll be using ActiveMQ as the message broker. When you use ActiveMQ, you need to add some client libraries into the /repository/components /lib directory. These libraries can be found at link [4]. Following is a sample configuration of the JMS Message-store.

  
<messageStore class="org.apache.synapse.message.store.impl.jms.JmsStore" name="JMSMS">
   <parameter name="java.naming.factory.initial">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
   <parameter name="java.naming.provider.url">tcp://localhost:61616</parameter>
   <parameter name="store.jms.JMSSpecVersion">1.1</parameter>
</messageStore>

Step 2 - Adding Store mediator

Now that we have created the Message-store, let’s create a proxy service that stores messages using the above message store. In order to store messages, we need to use store-mediator as shown in the configuration below.

  
<proxy name="SimpleProxy" transports="http https" startOnLoad="true" trace="disable">
   <target>
      <inSequence>
         <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2" type="STRING"></property>
         <property name="OUT_ONLY" value="true" scope="default" type="STRING"></property>
         <store messageStore="JMSMS"></store>
      </inSequence>
   </target>
</proxy>

Step 3 - Adding Message processor

Lastly, let’s create a Message-processor that consumes messages from the the message-store and sends it to an endpoint. For this example, we use a Forwarding-Message-Processor, which is always bound to an endpoint. The following synapse configuration has an endpoint and a Forwarding-Message-Processor that uses this endpoint.

 
<endpoint name="sampleEndpoint">
   <address uri="https://localhost:9000/services/SimpleStockQuoteService" />
</endpoint>

<messageProcessor class="org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor" name="Forwarder" targetEndpoint="sampleEndpoint" messageStore="JMSMS">
   <parameter name="client.retry.interval">1000</parameter>
   <parameter name="interval">1000</parameter>
   <parameter name="is.active">true</parameter>
</messageProcessor>

This concludes all the basic configuration of all three constructors of MSMP. You can try out these configurations by simply copying them into source view of WSO2 ESB. Details of setting up the environment, such as adding client libraries, starting ActiveMQ, and backend service can be found at link [5] and these are not explained in this article as it is not the primary focus.

Note: Something that has to be highlighted is that when you set up MSMP for JMS brokers, you do not need to enable JMS transport of the ESB. Implementation of JMS transport and MSMP are totally independent from one another.

Implementation of MSMP

ESB 4.8.0 comes with a brand new implementation of MSMP. Starting from design to implementation, everything is changed. This section explains the problems of the predecessors of ESB 4.8.0 and how they were handled in the new implementation of ESB 4.8.0. Therefore, this section is more useful for developers of MSMP rather than users.

Approach of the implementation

According to the previous design of MSMP, most of the work is done by the MessageStore class. It was implemented with basic queue operations, such as put, peek, poll, etc. and acted as a proxy to the actual message stores, such as in-memory-maps and JMS-Queues. However, this design didn't cope well with scenarios, such as reliable messaging, e.g. we came across a lot of integration issues while we were integrating ESB with WSO2-MB.

This prompted us to shift from a queue-centric implementation to a producer/consumer style implementation. This makes it easier to implement the requirements associated with store-and-forward concepts, such as reliable messaging and throttling. According the new design, MessageStore class act as a Manager and provides consumers and producers as required to message processors and mediators in contrast to providing methods to do queue operations.

Figure 2 shows how the components are organized in the new implementation.

Figure 2

Figure 2 is aimed at showing a developer the bigger picture on how the core classes are organized and associated. There are a few more other classes that are used around these core classes that are not shown in the diagram; however, a developer can use this to get started.

Main technologies used

There are two main technologies that were used while implementing MSMP. As you may have already noticed, one is Java Messaging Service (JMS). Of course, this is only used to implement JMS message- store. However, JMS message-store is the message-store type that is widely used in production. Therefore, it is very important to understand the concepts of JMS, such as supported message types, acknowledgment types, and session management.

Moreover, in order to archive reliable messaging in JMS message-store, we have used CLIENT_ACKNOWLEDGMENT. Basically, what happens here is control of the acknowledgment is passed to the client application. In our case, we delay the acknowledgment of the consumed message until we get a proper response from the backend, which ensures reliable delivery of messages.

The other technology that we have used is the Quartz job scheduler. This is an open source project that is predominantly focused on job scheduling. Massage-processors are implemented around Quartz’s job scheduling. The main reason to choose Quartz job scheduler over java scheduler executors is Quartz’s supports for cron-expressions. For more information about Quartz refer to link [6].

State machine of Message Processor

Another piece of information that would help a developer while working on MSMP is the state machine of Message-processor. Figure 3 shows the available states of message-processor and how the switching between these states is done.

Figure 3

If you’re wondering if there is a difference between stop and pause state, the answer is yes; the difference between these two states is that when the message-processor is in stop state, it simply does not do anything, but when it is in pause state, it stops consuming messages from the queue, but keeps on retrying to send the message that it has consumed already to the backend.

Improvements and future work

• Though current implementation of MSMP is robust and production ready, there is always room for improvements - those that could take the current implementation to the next level. Some improvement that we’ve toyed with are as follows:
• Improve the implementation of in-memory message-store - by now you may already know that current the implementation of in-memory message-store is at a primitive stage. It is possible to improve this by incorporating features, such as massage expiration, support persistent of messages, etc.
• Support other implementation such as AMQP (RabbitMQ, Apache Qpid) - there could be a situation where the user is already using an AMQP message broker. But since MSMP does not support AMQP message brokers, the user has to deploy another JMS message broker for the sake of MSMP. We can avoid this additional overhead on effort of configuration and resources by implementing MSMP to support protocols, such as AMQP.
• Dynamic creation of message-stores and message-processors - as of now, we could not create Message-stores and Message-processors at run time. There could be scenarios where the required message-stores and message-processors are not known at the time of deployment. Therefore, such requirements are not covered with the current implementation of MSMP.