Implementing libcurl based HTTP Transport in Apache Axis2/C

Introduction

cURL is a command line tool used for URL manipulation. 'libcurl' is the API that facilitates using cURL inside our programs. There are many language bindings for libcurl, for instance PHP, PERL, Ruby, etc,. The advantage of giving HTTP transport level manipulation to such a library is precise and efficient handling of the protocol.

'libcurl' Integration to Apache Axis2/C

cURL transport is more focused on Apache Axis2/C client side request handling. It (libcurl) is responsible for sending SOAP messages through the HTTP protocol.We can provide the necessary information for cURL, such as HTTP headers, but we still need to have control over the protocol. You will understand how to address these issues while reading further.

Install libcurl

Before building Axis2/C with libcurl, you need to make sure that libcurl is installed on your system.

Build with libcurl Support

./configure --enable-libcurl=yes [other options]

Linux

This option (--enable-libcurl) builds the system (Apache Axis2/C) with AXIS2_LIBCURL_ENABLED defined. The Apache Axis2/C HTTP transport sender will decide what to use based on this definition.

MS Windows

Set the following options in build/configure.in file

ENABLE_LIBCURL= 1
LIBCURL_BIN_DIR = "path to libcurl bin directory"

Using libcurl

typedef struct axis2_libcurl
{
   axis2_char_t *memory;
   axis2_array_list_t *alist;
   unsigned int size;
   const axis2_env_t *env;
}axis2_libcurl_t;

libcurl based HTTP transport implementation of Axis2/C is done by using the libcurl's 'easy' interface. The axis2_libcurl_t structure's 'memory' element is used for holding a response, and the 'size' element is used to hold its (memory element) size.

curl_easy_setopt () is used to set various options and curl_easy_perform () is used to perform the operation (send request and get response).

libcurl Options Used

1. cURL Initialization

Before you do anything with cURL, it's essential to initialize it.

    CURL *handler;
    handler = curl_easy_init ();
  

The initialized easy handler is used to set various options and perform the operation.

2. HTTP Header Manipulation

In libcurl, the user can manipulate the HTTP header through a structure called curl_slist .

struct curl_slist *headers = NULL;

If you need to add or remove HTTP headers, you can use curl_slist_append. The addition of headers can be done as follows.

        headers = curl_slist_append (headers, "User-Agent:Axis2/C");

By default, cURL sends some HTTP headers but Axis2/C doesn't have any special use of them. You can remove them as follows,

        headers = curl_slist_append (headers, "Accept:");
        headers = curl_slist_append (headers, "Expect:");

Once you have the necessary headers in hand, you need to set them to easy handler using curl_easy_setopt

curl_easy_setopt (handler, CURLOPT_HTTPHEADER, headers);

HTTP Headers Used

• HOST

The HOST HTTP header is one of the headers that libcurl uses. We also need to have this in our request. Therefore, we keep this header unchanged.

• User-Agent

For Axis2/C, the user agent should be "Axis2/C". We set this header in a single line.

     headers = curl_slist_append (headers,"User-Agent:Axis2/C");

• Content-Type

The content type differs based on the SOAP or REST request, SOAP1.1 or SOAP 1.2, MTOM enabled or disabled, and the char set encoding. Accordingly, we create the 'content-type' header string and append it to the 'curl_slist' that we have already created.

• Content-Length

  The buffer length of the SOAP is used as a content length. You can directly obtain the buffer length from the XML writer.

• SOAPAction

  SOAP Action is a mandatory header for SOAP 1.1 requests. After determining whether the request is SOAP 1.1, you can obtain the SOAP action from the message context. (The message context is a context that holds certain properties about a message.)

• HTTP Body/Content Management

The SOAP request that we got from the XML parser has to be handed over to the HTTP request. This is done by using CURLOPT_POSTFIELDS and the size of the buffer is set as CURLOPT_POSTFIELDSIZE.

    curl_easy_setopt (handler, CURLOPT_POSTFIELDSIZE, buffer_size);
    curl_easy_setopt (handler, CURLOPT_POSTFIELDS, buffer);
  

• URL

  curl_easy_setopt (handler, CURLOPT_URL, str_url);

  The URL of the endpoint is set as above.

  NOTE:

  When handling HTTP GET, it is essential to append some data fields to a URL and use 'url encode' for a safe transfer. Still libcurl does not handle it as a part of its job. Therefore, we need to handle that scenario using a third party library. Apache Axis2/C uses an internal function to send a REST request through HTTP GET.

• WRITEFUNCTION

Since we have to read the response data to a buffer, we must register a callback in libcurl in the following format.

size_t function( void *ptr, size_t size, size_t nmemb, void *stream);

In Axis2/C, we use a callback as follows.

    size_t 
    axis2_libcurl_write_memory_callback(
    void *ptr, 
    size_t size, 
    size_t nmemb, 
    void *data);

  

"ptr" is the data pointer given by the cURL. The size of that ptr is a product of the "size" to "nmemb". "data" is a structure that holds incoming data according to the cURL. This function is untouched. Register this function and the stream that is provided to hold data.

    curl_easy_setopt (handler,CURLOPT_WRITEFUNCTION, axis2_libcurl_write_memory_callback);
    curl_easy_setopt (handler, CURLOPT_WRITEDATA, data);

If we did not provide this callback, libcurl will write to stdout.

• HEADERFUNCTION

  As with the WRITEFUNCTION, if we intend to receive HTTP headers, we must register a callback in the following format,

  size_t function( void *ptr, size_t size, size_t nmemb, void *stream);

  using the CURLOPT_HEADERFUNCTION, CURLOPT_WRITEHEADER.

• DEBUG

  curl_easy_setopt (handler, CURLOPT_VERBOSE, 1);

  If you are doing any development using libcurl, you should be aware of this function. When CURLOPT_VERBOSE is set to 1, it prints details about the operations it carries out.

  Performing various functions with libcurl in Axis2/C is all about setting the applicable options. After setting the options, you can perform the sending task.

  curl_easy_perform (handler);

axis2_libcurl_stream Implementation

There has to be a mechanism to read data on request. Although we have response data on hand, in order to have a smooth integration, the libcurl stream implementation is required. Since Apache Axis2/C has an extensible architecture, there is not much work to be done to plug in the stream implementation.

To implement the libcurl stream, the interface available in axis2_stream.h is used. After the implementation, the stream is set to the message context as a property named AXIS2_TRANSPORT_IN. Later, this stream is set to a callback, which is used to build the response SOAP message.

Conclusion

Handling HTTP protocols through a library like 'libcurl' has the advantage of fine tuned and precise communication. It is a widely used and well tested transport library. It has many HTTP protocol handling features like HTTP-Chunking and IPV6. Apache Axis2/C will be able to gain these advantages by using this implementation.

References

1. Apache Axis2/C
2. cURL

Author

Dinesh Premalal, Senior Software Engineer, WSO2 Inc. dinesh at wso2 dot com