Version: 1.0-beta

ELK Stack: Complete Log Analytics

Complete log aggregation, full-text search, and visualization using the Elastic Stack. Perfect for organizations requiring advanced log analytics, compliance auditing, and long-term log retention.

Architecture

WSO2 Agent Builder agent
├── Logs (stderr) ──▶ Filebeat ──▶ Logstash ──▶ Elasticsearch ──▶ Kibana
│                                   (optional)      (indexing)       (UI)
│
└─ Metrics (9797) ──▶ Prometheus ──▶ Grafana (optional, for metrics)

Prerequisites

Docker & Docker Compose
4GB RAM minimum (8GB recommended)
20GB disk space for Elasticsearch data

Docker Compose Setup

Create docker-compose.yml. The agent service runs the executable JAR built from your agent project (placed in an agent directory next to this file) and writes its logs in JSON format so that each field is indexed:

services:
  # Elasticsearch (Log Storage & Search)
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - xpack.security.enrollment.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    networks:
      - elk

  # Kibana (Visualization)
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Logstash (Log Processing)
  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    ports:
      - "5044:5044"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    environment:
      - xpack.monitoring.enabled=false
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Filebeat (Log Shipper)
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    command: filebeat -e -strict.perms=false
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - logstash
    networks:
      - elk

  # WSO2 Agent Builder agent
  agent:
    image: eclipse-temurin:21-jre
    working_dir: /app
    command: sh -c "java -jar target/bin/*.jar"
    ports:
      - "8090:9090"
      - "9797:9797"
    environment:
      - BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSENABLED=true
      - BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSREPORTER=prometheus
      - BAL_CONFIG_VAR_BALLERINAX_PROMETHEUS_PORT=9797
      - BAL_CONFIG_VAR_BALLERINA_LOG_LEVEL=INFO
      - BAL_CONFIG_VAR_BALLERINA_LOG_FORMAT=json
    volumes:
      - ./agent:/app
    networks:
      - elk

  # Prometheus (Optional, for metrics)
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - elk

  # Grafana (Optional, for metrics visualization)
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    depends_on:
      - prometheus
    networks:
      - elk

volumes:
  elasticsearch-data:
  prometheus-data:
  grafana-data:

networks:
  elk:
    driver: bridge

Configuration Files

Agent configuration

Enable observability in the agent's Ballerina.toml:

[build-options]
observabilityIncluded = true

Import the Prometheus extension in one of the agent's .bal files:

import ballerinax/prometheus as _;

Filebeat Configuration (`filebeat.yml`)

Filebeat discovers the agent container through the Docker socket and ships only that container's logs:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.labels.com.docker.compose.service: agent
          config:
            - type: filestream
              id: agent-${data.docker.container.id}
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              parsers:
                - container: ~
                - ndjson:
                    target: ""
                    overwrite_keys: true
                    add_error_key: true
              fields:
                service: wso2-agent
                environment: production
              fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info

Logstash Configuration (`logstash.conf`)

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "json" {
    json {
      source => "message"
    }
  }

  # Parse timestamp
  if [time] {
    date {
      match => ["time", "ISO8601"]
      target => "@timestamp"
    }
  }

  # Add service metadata
  mutate {
    add_field => {
      "log_source" => "ballerina-agent"
      "processing_timestamp" => "%{@timestamp}"
    }

    # Remove empty fields
    remove_field => ["[input][type]"]
  }

  # Copy the agent session ID, if present, into a conversation ID field
  if [sessionId] {
    mutate {
      add_field => { "conversation_id" => "%{sessionId}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "ballerina-agents-%{+yyyy.MM.dd}"
  }

  # Optional: Forward to stdout for debugging
  # stdout {
  #   codec => rubydebug
  # }
}

Prometheus Configuration (`prometheus.yml`)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "wso2-agent"
    static_configs:
      - targets: ["agent:9797"]

Grafana Datasources (`grafana-datasources.yml`)

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Running the Stack

# Build the agent
cd agent
bal build
cd ..

# Start all services
docker compose up -d

# Verify services are running
docker compose ps

# View logs
docker compose logs -f

Accessing the Services

Kibana: http://localhost:5601
Elasticsearch: http://localhost:9200
Grafana (metrics): http://localhost:3000
Prometheus: http://localhost:9090

Setting up Kibana data view

Open Kibana at http://localhost:5601
Go to Stack Management > Data Views
Click Create data view
Index pattern: ballerina-agents-*
Timestamp field: @timestamp
Click Save data view to Kibana

Creating Kibana Visualizations

1. Log Volume Over Time

Go to Visualize > Create Visualization
Select Area chart type
Elasticsearch index: ballerina-agents-*
X-axis: Date Histogram on @timestamp
Y-axis: Count
Add filter: level: "ERROR" (optional)

2. Error Rate Table

Create Data Table visualization
Elasticsearch index: ballerina-agents-*
Bucket aggregation: Terms on level.keyword
Metric: Count
Filter: level: "ERROR"

3. Service Breakdown Pie Chart

Create Pie visualization
Elasticsearch index: ballerina-agents-*
Bucket: Terms on module.keyword
Metric: Count

4. Custom Kibana Dashboard

Go to Dashboards > Create Dashboard
Add your visualizations
Search by session ID: sessionId: "session-42"
Search by log level: level: "ERROR" OR level: "WARN"
Search by module: module: "myorg/support_agent"

Useful KQL Queries

Query	Purpose
`level: "ERROR"`	Find all error logs
`service: "wso2-agent" AND sessionId: "session-42"`	Follow a specific conversation
`level: "ERROR" AND NOT message: "timeout"`	Errors excluding timeouts
`message: "Ticket summary failed"`	Search error messages
`sessionId: *`	Logs with a sessionId field
`level: "WARN" OR level: "ERROR"`	Warnings and errors

Advanced: Logstash Parsing Examples

These patterns extract values written into the log message text itself, not values passed as separate key-value pairs.

Extract Conversation Information

filter {
  if [message] =~ /Chat request received/ {
    grok {
      match => {
        "message" => "Chat request received.*sessionId=%{DATA:sessionId}.*customerId=%{DATA:customerId}"
      }
    }
  }
}

Extract API Response Times

filter {
  if [message] =~ /response_time/ {
    grok {
      match => {
        "message" => "API call.*response_time=%{NUMBER:response_time:float}ms"
      }
    }
    mutate {
      convert => { "response_time" => "float" }
    }
  }
}

Performance Tuning

Elasticsearch JVM Heap

# Edit docker-compose.yml
environment:
  - "ES_JAVA_OPTS=-Xms2g -Xmx2g"

Index Retention Policy

# Delete an old index by name (Elasticsearch rejects wildcard deletes by default)
curl -X DELETE "localhost:9200/ballerina-agents-2024.01.01"

Logstash Pipeline Config

# Optimize batch settings (logstash service in docker-compose.yml)
environment:
  - PIPELINE_BATCH_SIZE=500
  - PIPELINE_BATCH_DELAY=50

Troubleshooting

No logs appearing in Kibana:

# Check Filebeat is running
docker compose logs filebeat

# Check Logstash is receiving beats
docker compose logs logstash | grep "beats"

# Check Elasticsearch has data
curl http://localhost:9200/_cat/indices

Index pattern not found:

# Verify index exists
curl http://localhost:9200/_cat/indices?v

# Wait for data to arrive (can take 30 seconds)
docker compose logs agent | tail -20

High disk usage:

Reduce log retention: index => "ballerina-agents-%{+yyyy.MM}"
Delete old indices regularly
Increase Elasticsearch volume size

Cleanup

# Stop all services
docker compose down

# Remove all data
docker compose down -v

What's Next

Elastic Stack (ELK) — The focused integration guide for this platform
OpenSearch Setup – Open-source alternative
Logging Overview – Learn more about logging
Metrics Overview – Add metrics collection

Architecture​

Prerequisites​

Docker Compose Setup​

Configuration Files​

Agent configuration​

Filebeat Configuration (filebeat.yml)​

Logstash Configuration (logstash.conf)​

Prometheus Configuration (prometheus.yml)​

Grafana Datasources (grafana-datasources.yml)​

Running the Stack​

Accessing the Services​

Setting up Kibana data view​

Creating Kibana Visualizations​

1. Log Volume Over Time​

2. Error Rate Table​

3. Service Breakdown Pie Chart​

4. Custom Kibana Dashboard​

Useful KQL Queries​

Advanced: Logstash Parsing Examples​

Extract Conversation Information​

Extract API Response Times​

Performance Tuning​

Elasticsearch JVM Heap​

Index Retention Policy​

Logstash Pipeline Config​

Troubleshooting​

Cleanup​

What's Next​