2013/01/31
31 Jan, 2013

Streaming XPath Parser for WSO2 ESB

  • Andun Sameera
  • Software Engineering Intern - WSO2

What is the Streaming XPath Parser?

To select and obtain content from an XML document, we use the XPath (XML Path Language) query language. XPath plays a major role in SOA, since most of the messages going through the SOA environment are SOAP messages, which are a form of XML. To manipulate these SOAP messages, we use XPath expressions. We use XPath engines like Jaxen or AXIOM XPath to parse the expressions and select content from the XML file that matches the conditions specified by the expression.

For example, assume you use the expression "/data//book" to select content from the following XML:


  
       
   Andun
   Sri Lanka
       
  
  
       
		   Mr_Thilak_Fernando
		   
			   108
			   328
			   568
		   
      
      
		   Ms_Pivithuru
		   
			   88
			   328
			   568
			   808
			   1048
		   
      
      
		   Mr_Nisansa
		   
			   88
			   328
			   568
			   808
			   1048
		   
      
  


The given XPath expression "/data//book" specifies that all the “book” elements that are children of the “data” element should be returned. Therefore, the engine must open each child element of the “data” element, determine whether it is a “book”, and then return all the found book elements as a list. Navigation through each and every node consumes time and memory, so the larger the XML file, the more parsing performance is reduced, regardless of the number of actual matches found. Because performance is especially important for an ESB, we need an optimized approach to parsing XPath.

Fortunately, in an SOA environment the message format is usually fixed, so we can use an optimized parsing approach instead of searching every element in the XML document. In WSO2 ESB 4.6.0, we introduced the Streaming XPath parser, which uses a special optimization designed to handle the common use cases in an ESB environment. This feature is build on top of the Pass Through Transport of the WSO2 ESB.

Following are the key characteristics of the Streaming XPath parser.

  1. The XPath parsing happens on top of the AXIOM data model created by an input stream to the XML document
  2. The advantage of this approach is that AXIOM will not consume the unwanted part of the steam, thereby reducing the memory and time taken to open an XML document as a Java object model.

  3. Returns the first match of the XPath expression
  4. For example, assume we use an expression like "/data/book/author/name” against the following XML:

    
    	
    		
    		Andun
    		Sri Lanka
    	
    	
    	
    		
    		Sameera
    		Sri Lanka
    		
    	
    
    

    The first author element is returned as the result of the expression. This approach is useful for an ESB environment, because most of the time the use case is to get the first match. XPath expressions like “/data//book” will also result in the first match when the Streaming XPath parser is used. For example, take a look at the following XML against the expression “/data//book”:

    
    	
    		
    		Andun
    		Sri Lanka
    	
    	
    	
    		
    			
    			Sameera
    			Sri Lanka
    			
    		
    	
    
    

    The result of the given XPath will be the first book element. The book element inside the book-rack element will be not in the result.

    This approach saves the time it would take to search all over the XML tree to get a result, and it reduces the memory needed to do the parsing, since we don’t traverse the rest of the document after we have found the first result.

  5. Complex XPath expressions that have predicates, logical combinations of results, etc. are not eligible for Streaming XPath

Component-based Architecture of the Streaming XPath Parser

The implementation of Streaming XPath uses a component-based architecture. The first thing the parser does is create an AXIOM data model of the XML document using the given input stream to the XML. Using the data model, the parser finds the result for a given XPath expression.

There is a set of components (Java classes) that are responsible for performing simple operations on the AXIOM data model. These components are like a component of a processing chain in a manufacturing plant. Each component will take an input (here it is an OMElement) and will output a String as a result. For example:

  1. Component to return an XML node if it matches some conditions like QName, Position, etc.
  2. Component to return a set of children of an XML node if it matches some conditions like QName, Position, etc.
  3. Component to return an attribute of an XML node

Using those components, we are creating a mechanism to find the result for a given XPath expression. The following diagram illustrates how those components are used to parse an XPath expression.

Streaming XPath Componets Chain

Here the AXIOM data model for the XML document will be passed to the first component, which will choose the first element that has the local name “data”. It will then send that element to the next component to select the book child of that element. Likewise the last component will output the result of the XPath expression.

In summary, here's what happens inside the Streaming XPath parser:

  1. When we give an XPath expression to the parser, it creates a component chain like above, which will do the processing to get the result of the XPath.
  2. When the parsing happens, the input XML is passed though the component chain and output is obtain at the end of the chain as the result.

We support a limited set of XPath expressions in the Streaming XPath parser. We have a set of components like above, which are used to find the result of XPath expressions. If you wanted to add support for more complex XPath expressions, you would have to add such a component to the parser.

How is the XML processing component chain created?

The Streaming XPath parser supports simple absolute and relative XPath expressions that don't have logical combinations, functions, predicates (other than predicate [1]), etc. When an XPath expression is given, the Streaming XPath parser will analyze that expression to check whether it belongs to the above set of expressions. If it matches, it will create the component chain described above.

For that matching and component chain creating process, we use a language-analyzing tool called Antlr. Using Antlr, we check whether a given string matches a particular syntax, or set of rules, which is provided in a grammar file. We have created a grammar file called XPath1.g, which contains all the rules of XPath that we support. Using that grammar file we analyze XPath expressions. So we can choose the XPath expressions that can be parsed via Streaming XPath.

When a given XPath expression matches, the Antlr parser will generate a syntax tree for that expression. The rules used to generate the syntax tree are provided in a grammar file called XPathWalker.g. The following diagram shows such a syntax tree that is generated for an XPath expression:

Abstract Syntax Tree of XPath

Antlr provides the functionality to generate Java code according to the structure of the syntax tree. The code parts to be included are specified in the XPathWalker.g file. Antlr will merge all the code parts into a Java class. We create the component chain using the tree generated for the XPath expression, so we have added code segments to the rule set to create the competent chain at the end of analyzing the XPath expression.

Following are the Java components in the Streaming Parser:

  1. GetAttributeParserComponent
  2. GetChildrenByNameParserComponent
  3. GetChildrenByNameRelativeParserComponent
  4. GetCurrentMatchParserComponent
  5. GetCurrentParserComponent
  6. GetFirstChildParserComponent
  7. GetChildrenWithAttributeValueParserComponent
  8. GetChildrenWithAttributeParserComponent

We combine these components according to the syntax of the expression and the rules we have defined. For example, there is a rule to add a GetCurrentParserComponent to the component chain if we have a “.” in the XPath expression. Also, there is a rule to add a GetChildrenByNameParserComponent to the component chain if we have a “…./some name/…” in the XPath expression.

In summary, we analyze an XPath expression as follows:

  1. Get an XPath expression and match it against the ruels of XPath1.g using Antlr
  2. Generate the syntax tree using the rules in XPathWalker.g file if the XPath expression matches the rules
  3. If the expression doesn't match the rules, Antlr throws exceptions
  4. Generate the Java code for the syntax tree using Antlr

Merging of Streaming XPath with Synapse XPath

In WSO2 ESB 4.6.0, the Synapse XPath and Streaming XPath are merged. But the Streaming XPath parsing is enabled on user request only. To enable Streaming XPath, you must set the following property in the Synapse Properties file:

synapse.streaming.xpath.enabled=true

The constructor of the Synapse XPath class checks whether Streaming XPath is enabled through this property. If yes, it will analyze the given XPath expression using Antlr. If the expression matches the conditions, the Streaming XPath Parser is initialized. If the expression doesn’t match the conditions, the default Synapse XPath Parser is initialized. The following code shows how this is done in the constructor:

private String enableStreamingXpath = SynapsePropertiesLoader.loadSynapseProperties().getProperty(SynapseConstants.STREAMING_XPATH_PROCESSING);
if("true".equals(enableStreamingXpath)){
    try {
        this.streamingXPATH = new StreamingXPATH(xpathString);
        contentAware = false;
    } catch (StreamingXPATHException e) {
        if (log.isDebugEnabled()) {
            log.debug("Provided XPATH expression " + xpathString + " cant be evaluated Streaming XPath.");
        }
        contentAware = true;
    } catch (StreamingXPATHCompilerException exception) {
        if (log.isDebugEnabled()) {
            log.debug("Provided XPATH expression " + xpathString + " cant be evaluated Streaming XPath.");
        }
        contentAware = true;
    }
}

When a message comes in while the Streaming XPath parser is enabled, an input stream to the incoming message is sent to the parser, and the result will be obtained. Otherwise, default Synapse XPath is executed. The following code shows how this is done:

InputStream inputStream = null;
Object result = null;
org.apache.axis2.context.MessageContext axis2MC =null;

if ("true".equals(enableStreamingXpath)&& streamingXPATH != null) {
    try {
        axis2MC = ((Axis2MessageContext)synCtx).getAxis2MessageContext();//((Axis2MessageContext) context).getAxis2MessageContext();
        inputStream=getMessageInputStreamPT(axis2MC);
    } catch (IOException e) {
        e.printStackTrace();
    }
    if (inputStream != null) {
        try {
            result = streamingXPATH.getStringValue(inputStream);
        } catch (XMLStreamException e) {
            handleException("Error occurred while parsing the XPATH String", e);
        } catch (StreamingXPATHException e) {
            handleException("Error occurred while parsing the XPATH String", e);
        }
    } else {
        try {
            result = streamingXPATH.getStringValue(synCtx.getEnvelope());
        } catch (XMLStreamException e) {
            handleException("Error occurred while parsing the XPATH String", e);
        } catch (StreamingXPATHException e) {
            handleException("Error occurred while parsing the XPATH String", e);
        }
    }
} else {
	//Default Synapse XPath
    result = evaluate(synCtx);
}

You can further look at the code of the Streaming XPath implementation here.

Conclusion

The ESB-specific customization and optimization of XPath parsing through the Streaming XPath parser provides a huge impact on performance. It has doubled the performance of the XPath evaluation process compared to the previous implementation.

Author

Andun S.L. Gunawardana, Trainee Software Engineer , WSO2 Inc.

References

 

About Author

  • Andun Sameera
  • Software Engineering Intern
  • University Of Moratuwa