Elastic Stack (ELK)

The Elastic Stack (Elasticsearch, Logstash, Kibana) provides powerful log aggregation, search, and visualization for your WSO2 Agent Builder agents. Use it to centralize logs, analyze conversation patterns, and create operational dashboards.

Prerequisites

Requirement	Details
Elasticsearch	Version 8.x or later
Logstash	Version 8.x (optional, Filebeat can replace it)
Kibana	Version 8.x
Filebeat	Version 8.x (recommended for log shipping)

Architecture

Ballerina Agent ──▶ Log Files ──▶ Filebeat ──▶ Elasticsearch ──▶ Kibana
                                            ▲
                   (optional) ──▶ Logstash ─┘

Step 1 -- configure Ballerina logging

Enable structured JSON logging to a file in Config.toml:

[ballerina.log]
level = "INFO"
format = "json"

[[ballerina.log.destinations]]
type = "file"
path = "/var/log/agents/app.log"

Emit structured log entries in your code. The following service hosts a support agent and logs each chat request and response:

import ballerina/ai;
import ballerina/http;
import ballerina/log;

// Default model provider
final ai:Wso2ModelProvider wso2ModelProvider =
    check ai:getDefaultModelProvider();

// Agent declaration
final ai:Agent supportAgent = check new (
    systemPrompt = {
        role: string `Support Agent`,
        instructions: string `You answer customer questions about
            orders, deliveries, and returns.`
    },
    model = wso2ModelProvider
);

service /api on new http:Listener(9090) {
    resource function post chat(@http:Payload ai:ChatReqMessage request)
            returns ai:ChatRespMessage|error {
        log:printInfo("Chat request received",
            sessionId = request.sessionId,
            channel = "api",
            agent = "support"
        );

        string answer = check supportAgent.run(request.message, request.sessionId);

        log:printInfo("Agent responded",
            sessionId = request.sessionId,
            status = "success"
        );

        return {message: answer};
    }
}

Step 2 -- install and configure filebeat

Install Filebeat to ship logs to Elasticsearch:

# Install Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-linux-x86_64.tar.gz
tar xzvf filebeat-8.11.0-linux-x86_64.tar.gz

Configure filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/agents/*.log
    json.keys_under_root: true
    json.add_error_key: true
    fields:
      service: support-agent
      environment: production

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "ballerina-agents-%{+yyyy.MM.dd}"

setup.template.name: "ballerina-agents"
setup.template.pattern: "ballerina-agents-*"

setup.kibana:
  host: "http://kibana:5601"

Start Filebeat:

./filebeat -e

Step 3 -- configure Logstash (Optional)

For advanced log processing, use Logstash between Filebeat and Elasticsearch. Save this pipeline as logstash.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  date {
    match => ["time", "ISO8601"]
    target => "@timestamp"
  }
  mutate {
    add_field => { "pipeline" => "ballerina-agent" }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "ballerina-agents-%{+yyyy.MM.dd}"
    action => "create"
  }
}

Then replace the output.elasticsearch section of filebeat.yml with:

output.logstash:
  hosts: ["logstash:5044"]

When you use the Docker Compose setup below, add a Logstash service to it:

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

Step 4 -- create Kibana dashboards

Data view

1. Open Kibana and navigate to Stack Management > Data Views.
2. Create a data view with the index pattern ballerina-agents-*.
3. Select @timestamp as the timestamp field.

Useful visualizations

Visualization	Type	Description
Log Volume	Area chart	Log entries over time by level
Error Logs	Data table	Recent ERROR-level log entries
Service Map	Pie chart	Log distribution by service
Response Times	Line chart	Average response duration over time

Sample KQL queries

Query	Purpose
level: "ERROR"	Find all error logs
fields.service: "support-agent" AND sessionId: "session-42"	Follow a specific conversation
level: "ERROR" AND NOT message: "timeout"	Errors excluding timeouts

Docker compose setup

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    command: filebeat -e -strict.perms=false
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/log/agents:/var/log/agents:ro
    depends_on:
      - elasticsearch

volumes:
  es-data:

What's next

• ELK Stack: Complete Log Analytics — A full Docker Compose setup with dashboards, queries, and tuning
• OpenSearch -- Alternative open-source search and analytics
• Logging overview -- Configure Ballerina logging
• Observability Overview -- Full observability architecture