Understanding Axis2/C Threading Model

Introduction

In a multi-threaded environment it is important to understand how the libraries you use behave in respect to that environment. If your libraries use their own threads, then it is important to understand them as well. Even if the library does not use its own threads, it is still important to know that library will not be used unexpectedly in your threaded environment. Axis2/C is a SOAP engine which can process SOAP messages when it is used as a SOAP server application. It can also be used as a tool to send SOAP messages from client side. If you use Axis2/C on your server side, you would most likely be using it to deploy your SOAP services, so it is important to understand the server side thread model of the Axis2/C engine.

Additionally, if you use Axis2/C as a SOAP message sender, then you should understand how it could be used in your client application, especially if your client is a threaded application. In this article, I will explain the thread-related behavior of Axis2/C so you can use it safely in both of the above use cases.

Table of Contents

• Axis2/C Threads
• Ways It Can Be Improved
• Summary
• Resources

Axis2/C Threads

Designing threads for programs written in the C language requires special attention in terms of portability. Thread API's could vary greatly from operating system to operating system. Hardware vendors have implemented proprietary versions of threads. These implementations are substantially different from each other, making it difficult for programmers writing in the C language to write portable programs. There arises the need for a common interface for thread programming in C. This requirement is fulfilled by IEEE POSIX 1003.1c standard (1995) with its introduction into the POSIX threads standard for Unix platforms. Most hardware vendors now implement POSIX in addition to their own proprietary API's.

However, in certain platforms where Axis2/C is used, there is a need for providing some other thread model. For example, Axis2/C wanted to support Microsoft Windows platform from the beginning. So thread API designed for Axis2/C has implementations basically as wrappers around POSIX threads and Windows threads.

The API files are axutil_thread.h, and axutil_thread_pool.h. It should be noted that currently these API files are intended to be used within Axis2/C library, except for the case that a client program could provide his own thread pool when creating the Axis2/C environment instance. If a developer needed to implement this API for a particular thread model in his own platform, he could learn by investigating how it is done for Windows and POSIX threads. The source for these resides in<Axis2/C source>/util/src/platforms/windows and <Axis2/C source>/util/src/platforms/unix.

In client side, threads are used in non-blocking communication. Non-blocking communication could be divided into two types:

1. Non-blocking communication using callback and separate listener.

2. Non-blocking communication using only a callback.

In the first approach we need to tell the Axis2/C engine to use a separate listener in addition to specifying a callback.

axis2_options_set_use_separate_listener(options, env, AXIS2_TRUE);

This would cause Axis2/C engine to start its client side listener manager to listen for responses in specified transports. Listener transports could be specified in the client Axis2/C configuration file as follows.

    <transportReceiver name="http" class="axis2_http_receiver">        
        <parameter name="port" locked="false">6060</parameter>
        <parameter name="exposeHeaders" locked="true">false</parameter>
        </transportReceiver>    
    <transportReceiver name="https" class="axis2_http_receiver">
        <parameter name="port" locked="false">6060</parameter>
        <parameter name="exposeHeaders" locked="true">false</parameter>
        </transportReceiver>
    <transportReceiver name="tcp" class="axis2_tcp_receiver">
         <parameter name="port" locked="false">6060</parameter>
    </transportReceiver>

In the above configuration, we see that client listener manager is configured to listen for responses in transports http, https and tcp. When a response arrives with a transport specified, then a new thread is spawned to process the response by the transport receiver. If the client programmer does not specify use separate listener option, but still wants to do non-blocking communication, he can do so by specifying just a callback. Then Axis2/C engine spawns a new thread to send the request, and immediately returns the control to the client program. This new thread sends the request, and awaits a response in the same thread. In this approach, if you don't specify a separate listener for each request, a socket is made to wait on IO. That would have adverse effects when there are large numbers of requests simultaneously.

In both of the above methods, when the response arrives, it is first processed by the Axis2/C handlers. Control is then passed to the client provided callback function to process the response message.

In both approaches, we need to set a callback as below:

    callback = axis2_callback_create(env);
    /* Set our on_complete function pointer to the callback object */
    axis2_callback_set_on_complete(callback, echo_callback_on_complete);
    /* Set our on_error function pointer to the callback object */
    axis2_callback_set_on_error(callback, echo_callback_on_error);
    /* Send request */
    axis2_svc_client_send_receive_non_blocking(svc_client, env,payload, callback);

Here, callback function echo_callback_on_complete() is a user defined function which is used to handle the response for the request. This function is called by Axis2/C engine when callback returns on completion and response is available.

It has the signature:

axis2_status_t AXIS2_CALL    
    your_callback_on_complete_function(
                          struct axis2_callback * callback,
                          const axutil_env_t * env)

Callback structure has a field called data of type void pointer to store callback specific data. It also has a field to store the response SOAP envelope. You can also check its boolean field called 'complete', to check whether the callback is completed and result is available(by polling/busy waiting using axis2_callback_get_complete() function). You can also use the axutil_thread_mutex_t type field called mutex to protect your data within your callback function. You can also call axis2_callback_get_error() callback api function to get the error status after callback is completed.

When response to the request reaches Axis2/C engine and hits the client side message receiver then control is passed to this callback function, to process the response message as wanted by the client application developer.

If response is a soap fault then echo_callback_on_error() function is invoked instead of the above callback function.

There is another way you can use Axis2/C in a multi-threaded environment. This is by calling Axis2/C service clients or code generated stubs synchronous API functions to send/receive messages from your threads. The trick here is that you handle the asynchronous behavior in your own application program instead of relying on service client's/stubs asynchronous API functions.The idea is, when you need to send/receive messages in multiple threads, you create service clients/stubs for each thread. Next, you send messages using,

AXIS2_EXTERN axiom_node_t *AXIS2_CALL
    axis2_svc_client_send_receive(
                       axis2_svc_client_t * svc_client,
                       const axutil_env_t * env,
                       const axiom_node_t * payload);

In this way, you can use your own controlled thread pool, instead of relying on Axis2/C internal thread pools.  However, there are currently flaws in the Axis2/C internal thread mechanism (which will be discussed, along with possible solutions, in the next section).

This method is a little bit expensive as you need to create/initialize the environment and service client for each request to avoid conflicts. It should be noted that initializing service client means reading the main configuration file and populating main Axis2/C configuration context for each call. This is expensive. However, you can reuse this main configuration context by creating the service client or stub by passing it as a parameter.

For service client,

    AXIS2_EXTERN axis2_svc_client_t *AXIS2_CALL
        axis2_svc_client_create_with_conf_ctx_and_svc(
            const axutil_env_t * env,
            const axis2_char_t * client_home,
            axis2_conf_ctx_t * conf_ctx,
            axis2_svc_t * svc);

It should be noted that currently the stub API does not have an API function to create the stub, passing a configuration context as parameter. However, this will be available in the near future.

So far, we discussed how the thread model works at client side. On the server side, an incoming request is served by a new thread spawned by the server transport listener. It is within this thread that the service written by the service programmer functions.

Ways It Can Be Improved

It should be noted that there are still much needed improvements to the Axis2/C thread model. I'll discuss here some of the possible improvements which may be available in future versions of Axis2/C.

Every Axis2/C API function need to be passed the axutil_env_t environment struct pointer which contain important configuration structs for Axis2/C. In your client program you need to create this struct and pass it when you call service client API functions. If you create this just by passing the allocator for your environment then other structs like error, log and thread pool are created automatically by using Axis2/C default configuration structs. However if you need to override the default environment you can pass your own error handling struct, logging struct and thread pool struct using the following API function.

AXIS2_EXTERN axutil_env_t *AXIS2_CALL
        axutil_env_create_with_error_log_thread_pool(
        axutil_allocator_t * allocator,
        axutil_error_t * error,
        axutil_log_t * log, 
        axutil_thread_pool_t * pool);

However it should be noted that currently Axis2/C doesn't make use of the thread pool facility even if you pass one. It just requests threads from the thread pool and may exhaust all available threads until blocking your main thread at some point which is not what you desire by passing a thread pool. This limitation may be addressed in a future version not by requesting threads from the passed thread pool but by passing the job to the thread pool so that passed thread pool handles the jobs without ever blocking the main thread.

Another improvement would be to implement a new transport mechanism using asynchronous IO as in Boost library[2] where one thread handle multiple sockets for asynchronous communication and use callbacks to inform IO completion. Boost library uses Proactor IO design pattern.

Summary

It is important to know that in the Axis2/C thread model, portability is achieved using a portable thread API design. There are several important aspects of using multiple threads in your client environments for asynchronous calls that you need to know to be successful. While this is a very useful tool, there is still room for improvement in the Axis2/C thread model.

Resources

1. Axis2/C- Axis2/C is an Apache Web Services Platform implemented in C language around which the WSO2 WSF/C is built.
2. Boost- Boost.Asio is a cross-platform C++ library for network and low-level I/O programming that provides developers with a consistent asynchronous model using a modern C++ approach.

Author

Damitha Kumarage, Senior Software Engineer at wso2, committer Apache Software Foundation, damitha at wso2 dot com