[Article] How to Achieve Delivery Reliability with Dead Letter Channel Pattern - Part 2

Table of contents

• Applies to
• Introduction
• Setting up WSO2 Enterprise Service Bus
• Loading ESB with additional jar files
• ESB configuration file changes
• Mediation logic implementation
• Testing the setup
• Message delivery verification
• Making sure all ESB nodes are active
• Message failure scenario (mediation error/BE failure)
• ESB node failure
• MB node failure
• Increasing performance
• JMS tuning
• Using caching
• Extended test case
• Conclusion
• Appendix 1
• Appendix 2

Applies to

Introduction

As described in the previous article, a WSO2 Enterprise Service Bus (WSO2 ESB) cluster and a WSO2 Message Broker (WSO2 MB) cluster need to be setup. Setting up a WSO2 MB cluster is straightforward and the necessary broker side configurations were described in the previous article. We will now move on to explain how to configure WSO2 ESB for JMS messaging. In addition we will discuss how to test the setup, how delivery reliability works in case of a node failure and some tuning parameters or improvements you can do to achieve higher performance on message delivery. It will also suggest some extended use cases that you can try out.

Setting up WSO2 Enterprise Service Bus

Loading ESB with additional jar files

To communicate with WSO2 MB, WSO2 ESB needs client java libraries. Navigate to the <MB_HOME>/client-lib folder and copy the following jar files to each node of the ESB cluster. Jar files should be copied to the <ESB_HOME>/repository/components/lib folder.

• andes-client-3.0.1
• geronimo-jms_1.1_spec-1.1.0.wso2v1
• org.wso2.securevault-1.0.0-wso2v2

ESB configuration file changes

At each ESB node, make the following configuration file changes in addition to those done during clustering.

Enable JMS transport receiver

Navigate to the [ESB_HOME]/repository/conf/axis2/axis2.xml file and add the following configuration under Transport Ins (listeners).

 
  <transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
      <parameter name="myTopicConnectionFactory" locked="false">
         <parameter name="java.naming.factory.initial" locked="false">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter>
          <parameter name="java.naming.provider.url" locked="false">repository/conf/jndi.properties</parameter>
          <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">ConnectionFactory</parameter>
          <parameter name="transport.jms.ConnectionFactoryType" locked="false">topic</parameter>
      </parameter>
  </transportReceiver>

Enable JMS transport sender

Navigate to the [ESB_HOME]/repository/conf/axis2/axis2.xml file and add the following configuration under Transport Outs (senders).

 
   <transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender">
       <parameter name="default" locked="false">
           <parameter name="java.naming.factory.initial" locked="false">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter>
           <parameter name="java.naming.provider.url" locked="false">repository/conf/jndi.properties</parameter>
           <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">ConnectionFactory</parameter>
       </parameter>
   </transportSender>

Configure JMS JNDI properties

Navigate to the [ESB_HOME]/repository/conf/jndi.properties file and edit it as below.

• connectionfactory.ConnectionFactory - this will be the connectionfactory used by JMS transport listeners and senders. Here you need to configure the JMS failover connection string. Please refer to the “Deployment of Servers” section in part 1 of this article to learn how to configure this for each ESB node (failover order is different for each ESB node). Refer here for failover options. Following is an example failover string for ESB 1 with failover order MB1, MB2, MB3.

 
connectionfactory.ConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://192.168.2.2:5675?retries='5'&connectdelay='50';tcp://192.168.2.2:5676?retries='5'&connectdelay='50';tcp://192.168.2.2:5677?retries='5'&connectdelay='50''

• topic.stockQuoteTopic = stockQuoteTopic - this will be the topic to which you rout the messages. The form here is topic.[jndiName] = [physicalName]. We are using the JNDI name for logic implementation
  Comment out the default queue name in jndi.properties file as you won’t be using it

Mediation logic implementation

We recommend you to use WSO2 Developer Studio to develop artifacts for the WSO2 ESB. You can develop the following mentioned artifacts as a single .CAR file and deploy it to the management node of the ESB cluster. Then they will be automatically be deployed to all the worker nodes in the cluster.

Artifacts are described in the bottom top order so that they can be implemented in the following order.

JMS message sender proxy

This proxy will accept incoming HTTP messages, convert it into JMS messages and send it to the durable topic. Note how we have given the topic name to which the message needs to be routed.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="StockQuoteProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="custom">
            <property name="STATE" value="message is sent to topic"/>
         </log>
         <property name="OUT_ONLY" value="true"/>
         <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
      </inSequence>
      <outSequence/>
      <endpoint>
         <address uri="jms:/stockQuoteTopic?&transport.jms.DestinationType=topic"/>
      </endpoint>
   </target>
   <description/>
</proxy>

Let’s call this proxy “StockQuoteProxy”.

HTTP endpoints

In this example we will be invoking a service bundled with WSO2 ESB itself.

• Navigate to [ESB_HOME]/samples/axis2Server/src/SimpleStockQuoteService and run ant. This will build the service.
• Navigate to [ESB_HOME]/samples/axis2Server and run the sh axis2server.sh command. This will start an axis2 server with the above service.

The URL of the service will look like https://localhost:9000/services/SimpleStockQuoteService. You can store the end point configurations in the registry and refer to them in the mediation logic (dynamic endpoint). Read the documentation to see how to define a dynamic endpoint using WSO2 Developer Studio.

Let’s name this endpoint as “StockQuoteServiceEP”.

Error sequence

If a failure happens during message mediation or while sending a message to back-end services, you need to rollback the JMS transaction and tell the broker not to dequeue the message.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="Error_Squence">
   <log level="custom">
      <property name="STATUS" value="Mediation error. Rollaback JMS transaction..."></property>
   </log>
   <property name="SET_ROLLBACK_ONLY" value="true" scope="axis2"></property>
</sequence>

Let’s call this error sequence “Error_Squence”.

Mediation sequence

Let’s implement message mediation as an ESB sequence. The only mediation done in this example is invoking the above HTTP endpoint. While any complex mediation logic can be placed here, there is one thing to be noted. Every mediation element that processes, sends or receives the message, MUST do so in a blocking manner as you are using JMS transactions here. This means you cannot use send or call mediators to invoke the above HTTP endpoint. You can use the callout mediator as it invokes the back-end in a blocking manner. You need to understand that each thread should go through each element in mediation one after the other.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="Mediation_Sequence">
   <property name="FORCE_ERROR_ON_SOAP_FAULT" value="true" scope="default" type="STRING"></property>
   <callout serviceURL="https://localhost:9000/services/SimpleStockQuoteService" action="urn:getQuote">
      <source xmlns:s12="https://www.w3.org/2003/05/soap-envelope" xmlns:ns="https://org.apache.synapse/xsd" xmlns:s11="https://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"></source>
      <target xmlns:s12="https://www.w3.org/2003/05/soap-envelope" xmlns:ns="https://org.apache.synapse/xsd" xmlns:s11="https://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"></target>
   </callout>
   <log level="full">
      <property name="REPLY" value="REPLY_FROM_BE"></property>
   </log>
   <filter xmlns:ns="https://services.samples" source="fn: boolean(//ns:getQuoteResponse)" regex="true">
      <then>
         <log>
            <property name="STATUS" value="JMS Transaction committed"></property>
         </log>
      </then>
      <else>
         <sequence key="Error_Squence"></sequence>
      </else>
   </filter>
</sequence>

Note: You need to verify whether each and every BE invocation is successful with a response message validation logic. If the BE doesn’t return the response you expect, you need to specifically invoke “Error_Squence”. The property “FORCE_ERROR_ON_SOAP_FAULT” will enforce the error sequence if a BE returns a soap fault. If the BE is down (endpoint failure) or if a timeout happens during a BE invocation, the ESB will automatically invoke the fault sequence.

If you invoke multiple backends here, make sure each operation done by the BEs are idempotent. If the transaction fails to invoke the second BE, since the first BE has already been processed the first BE will be invoked again when the message is rollbacked and received again. For example, if the first BE does a DB record insert, there will be duplicates. In such circumstances, consider using the transaction mediator of WSO2 ESB to wrap multiple BE invocations to gather them into a transaction and make them atomic.

Let’s call this core sequence “Mediation_Sequence”.

JMS message receiving proxy

As you may have already noticed, we start a JMS transaction, followed by all logics described above, after a JMS message is received to the ESB from the topic. In order to listen to the JMS topic you need to write a JMS proxy as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="JMSTopicListenerProxy"
       transports="jms"
       startOnLoad="true"
       trace="disable">
   <description/>
   <target faultSequence="Error_Squence">
      <inSequence>
         <log level="custom">
            <property name="STATUS" value="Message received from topic"/>
         </log>
         <sequence key="Mediation_Sequence"/>
         <log level="custom">
            <property name="STATUS" value="Mediation is success"/>
         </log>
    <drop/>
      </inSequence>
      <outSequence/>
   </target>
   <parameter name="transport.jms.ContentType">
      <rules>
         <jmsProperty>contentType</jmsProperty>
         <default>text/xml</default>
      </rules>
   </parameter>
   <parameter name="transport.jms.ConnectionFactory">myTopicConnectionFactory</parameter>
   <parameter name="transport.jms.SessionAcknowledgement">CLIENT_ACKNOWLEDGE</parameter>
   <parameter name="transport.jms.DestinationType">topic</parameter>
   <parameter name="transport.jms.SessionTransacted">true</parameter>
   <parameter name="transport.jms.SubscriptionDurable">true</parameter>
   <parameter name="transport.jms.Destination">stockQuoteTopic</parameter>
   <parameter name="transport.jms.DurableSubscriberName">esbConnection-xx</parameter>
   <parameter name="transport.jms.CacheLevel">consumer</parameter>
   <parameter name="transport.jms.DurableSubscriberClientID">1</parameter>
</proxy>

• After receiving the message from a topic we need to set the content type
• The received message will be directed to “Mediation_Sequence”
• Note how topic name and connection factory is given as parameters
• The JMS subscription ID will be “esbConnection-xx”. As this artifact will be deployed to all the ESB nodes in the cluster they will all have this subscription ID
• The property “transport.jms.SubscriptionDurable” makes sure that the topic will be a durable one
• The property “transport.jms.SessionTransacted” being set to true will make sure the message flow will be treated in a JMS transacted way
• The property “transport.jms.CacheLevel” of the consumer will make sure the same connection/session will be utilized for all the messages. JMS commit/rollback will be performed for the session, so it’s important to consume message by message. You can increase the number of concurrent consumers, but make sure they DON’T share same session

We will call this JMS proxy “JMSTopicListenerProxy”.

Testing the setup

• Start the MB cluster first
• Then start the ESB cluster
• Deploy a CAR file with the above artifacts to the ESB cluster. As soon as .car is deployed you should see the following log line in all three MB nodes
  
  

  Figure 1

• Create a JMeter test with the following message to be delivered to the ESB cluster. The SOAP action should be urn:getQuote. Configure the load balancer fronting ESB to do a round robin message dispatch to the three ESB nodes. You can find how to configure HAProxy to do this using the information in our documentation. A sample haproxy.cfg and steps are included in appendix 1. Additionally, appendix 2 has a sample JMeter test invoking the HAProxy url.

<soapenv:Envelope xmlns:soapenv="https://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="https://services.samples" xmlns:xsd="https://services.samples/xsd">
   <soapenv:Header/>
      <soapenv:Body>
      <ser:getQuote>
         <!--Optional:-->
         <ser:request>
            <!--Optional:-->
            <xsd:symbol>WSO2</xsd:symbol>
         </ser:request>
      </ser:getQuote>
   </soapenv:Body>
</soapenv:Envelope>

• Invoke the JMeter test with a known number of samples (x).

Message delivery verification

In order to verify whether all messages sent are successfully delivered to the backend SimpleStockQuote service you will need to count the invoked lines.

Figure 2

You can write a service to count the incoming number of messages to easily count the requests and verify. A similar Web application can be found here.

You need to see that all ESB nodes are functioning (verify with logs), receiving messages and sending messages to different MB nodes.

Making sure all ESB nodes are active

Here all ESB nodes will be sending messages to the topic in the MB cluster. At the same time all ESB nodes will be receiving messages from the MB cluster. Thus, we should see all relevant logs in all three ESB consoles as shown below:

Figure 3

Message failure scenario (mediation error/BE failure)

Now shutdown the axis2 server you are running. At this point all the requests hitting the service is a failure. The flow of a failed message is shown below:

• Transaction is rolled back and WSO2 MB receives a negative-acknowledgment (NAK) signal.
• It schedules the message to be redelivered with the “redelivery” JMS header immediately.
• The WSO2 ESB receives the message again. This time if the transaction is successful the message will be removed from the topic under the subscription ID that the ESB uses. If not a NAK is again sent to WSO2 MB.
  
  

  Figure 4

• This procedure does not happen for ever. After 10 (by default) attempts of redelivery, the MB will place the message in Dead Letter Queue (DLQ). The number of times redelivery should happen can be configured at [MB_HOME]/repository/conf/broker.xml. Please edit the number in the <maximumRedeliveryAttempts> tae under amqp.
• When a message is in DLC, they do not come into the message flow again. You can view DLC messages in the MB console.


Figure 5

• As depicted in above image, there are three operations you can perform on DLC messages. You can delete them forever, restore them to the original queue or route them to another preferred queue registred on the WSO2 MB. This provides you with an extremely flexible way of handling failed messages.
• In this example, choose a failed message and restore it. Through the ESB logs, you will see that the message is received by the ESB again. If the axis2 server is down, it will retry 10 times and route again to DLC. If the server is up the message will be successfully consumed.

ESB node failure

Shutdown or kill ESB-1 in the ESB cluster when JMeter test is running. Now the load balancer will not direct traffic there. If the load balancer can handle it, there will be no transaction loss after that.

MB node failure

If a node in the MB cluster fails, the relevant ESB nodes will failover to a liver broker. To experience this, shutdown/kill MB-1 in the MB cluster. Now the ESB node which is connected to MB-1 will failover to MB-2 which is still live. Now if we stop MB-2 also, both ESB-1 and ESB-2 nodes will failover to MB-3.

Figure 6

Note that restarting MB-1 node will not make ESB-1 connect back to MB-1 as we are using connection/session/consumer caching. To use MB-1 again, you will have to restart ESB-1.

Increasing performance

JMS tuning

There are tuning parameters related to JMS that you can use in the ESB. Please refer to WSO2 ESB official documentation page here for more information.

Using caching

There are three levels of caching you can use when dealing with JMS proxies. These include

• Connection - cache the connection and reuse it when creating sessions/consumers per message
• Session - cache the connection and session and reuse the session when creating consumers per message
• Consumer - cache connection/session/consumer and reuse all of them without creating anything per message

Extended test case

We would now like to introduce the following message flows as an extended use case for the scenario described in this article:

• Write a data service using WSO2 DSS and expose a simple mySQL table as a service
• Mediation process will insert incoming data into the above table via a data service
• Use the JMeter test script to run a script inserting data (this should be inserted in a reliable way)
• Simulate a service outage/service malfunction at the DSS layer and verify when the defect is rectified that no message loss has occurred

Conclusion

In the world of messaging a broker is used as the middle ground for messages. When the component’s message receiving and process malfunctions, this integration pattern preserves the message. This article gave a step-by-step explanation on how to configure WSO2 ESB and WSO2 MB to build a robust messaging system which is highly availability. Apart from that advantage, WSO2 MB also provides scalability with increasing messaging requirements for a system. At the same time, we can’t maintain absolute message order in a distributed message broker system while keeping up with the performance. In that case WSO2 MB with proper system requirements in place, can deliver around 2000 messages per second for messages that are around 1 kilobyte in size. Thus, it is evident that the WSO2 platform can be utilized to build fast performing, scalable and distributed messaging systems.

Appendix 1

Following are the steps to configure HAProxy in MAC OS X.

• Issue the command on a terminal. It will install HAProxy on MAC.

            brew update
            brew install haproxy

• Create a file called haproxy.cfg with the following content. This will forward the requests coming from esb.wso2.com:80 to the ESB nodes.

frontend ft_wrk
    bind esb.wso2.com:80
    default_backend bk_wrk
 
backend bk_wrk
    balance roundrobin
    server node1 127.0.0.1:8280
    server node2 127.0.0.1:8281
    server node3 127.0.0.1:8282

• Run HAProxy with the following command

       
            sudo haproxy -f <path to above file>

Appendix 2

Following is the JMeter script used to test the scenario:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.8" jmeter="2.13 r1665067">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <Arguments guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments">
          <elementProp name="host" elementType="Argument">
            <stringProp name="Argument.name">host</stringProp>
            <stringProp name="Argument.value">wsf.cdyne.com</stringProp>
            <stringProp name="Argument.metadata">=</stringProp>
            <stringProp name="Argument.desc">Host of Webservice</stringProp>
          </elementProp>
        </collectionProp>
      </Arguments>
      <hashTree/>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Number of Users" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">600</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">5</stringProp>
        <stringProp name="ThreadGroup.ramp_time">0</stringProp>
        <longProp name="ThreadGroup.start_time">1375525852000</longProp>
        <longProp name="ThreadGroup.end_time">1375525852000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Soap Request" enabled="true">
          <boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
            <collectionProp name="Arguments.arguments">
              <elementProp name="" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.value">&lt;soapenv:Envelope xmlns:soapenv=&quot;https://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:ser=&quot;https://services.samples&quot; xmlns:xsd=&quot;https://services.samples/xsd&quot;&gt;&#xd;
   &lt;soapenv:Header/&gt;&#xd;
      &lt;soapenv:Body&gt;&#xd;
      &lt;ser:getQuote&gt;&#xd;
         &lt;!--Optional:--&gt;&#xd;
         &lt;ser:request&gt;&#xd;
            &lt;!--Optional:--&gt;&#xd;
            &lt;xsd:symbol&gt;WSO2&lt;/xsd:symbol&gt;&#xd;
         &lt;/ser:request&gt;&#xd;
      &lt;/ser:getQuote&gt;&#xd;
   &lt;/soapenv:Body&gt;&#xd;
&lt;/soapenv:Envelope&gt;</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain">esb.wso2.com</stringProp>
          <stringProp name="HTTPSampler.port">80</stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">services/StockQuoteProxy</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSamplerProxy>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">text/xml; charset=utf-8</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">SOAPAction</stringProp>
                <stringProp name="Header.value">urn:mediate</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
          <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
            <collectionProp name="Asserion.test_strings"/>
            <stringProp name="TestPlan.comments">Verify content in response</stringProp>
            <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
            <boolProp name="Assertion.assume_success">false</boolProp>
            <intProp name="Assertion.test_type">16</intProp>
          </ResponseAssertion>
          <hashTree/>
        </hashTree>
      </hashTree>
      <ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="false">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>false</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>false</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>false</xml>
            <fieldNames>true</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>true</responseDataOnError>
            <saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
            <hostname>true</hostname>
            <threadCounts>true</threadCounts>
            <sampleCount>true</sampleCount>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
      <ResultCollector guiclass="StatGraphVisualizer" testclass="ResultCollector" testname="Aggregate Graph" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>false</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>false</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>false</xml>
            <fieldNames>true</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>true</responseDataOnError>
            <saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
            <hostname>true</hostname>
            <threadCounts>true</threadCounts>
            <sampleCount>true</sampleCount>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
      <ConstantThroughputTimer guiclass="TestBeanGUI" testclass="ConstantThroughputTimer" testname="Constant Throughput Timer" enabled="true">
        <intProp name="calcMode">2</intProp>
        <doubleProp>
          <name>throughput</name>
          <value>600.0</value>
          <savedValue>0.0</savedValue>
        </doubleProp>
      </ConstantThroughputTimer>
      <hashTree/>
      <ResultCollector guiclass="TableVisualizer" testclass="ResultCollector" testname="View Results in Table" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>false</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
            <threadCounts>true</threadCounts>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
    </hashTree>
  </hashTree>
</jmeterTestPlan>