Version: 5.1

ELK Stack: Complete Log Analytics

Complete log aggregation, full-text search, and visualization using the Elastic Stack. Perfect for organizations requiring advanced log analytics, compliance auditing, and long-term log retention.

Architecture

WSO2 Integrator
├── Logs (stderr) ──▶ Filebeat ──▶ Logstash ──▶ Elasticsearch ──▶ Kibana
│                                   (optional)      (indexing)       (UI)
│
└─ Metrics (9797) ──▶ Prometheus ──▶ Grafana (optional, for metrics)

Prerequisites

Docker & Docker Compose
4GB RAM minimum (8GB recommended)
20GB disk space for Elasticsearch data

Docker Compose Setup

Create docker-compose.yml. The integrator service runs the executable JAR built from your integration project (placed in an integration directory next to this file) and writes its logs in JSON format so that each field is indexed:

services:
  # Elasticsearch (Log Storage & Search)
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - xpack.security.enrollment.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    networks:
      - elk

  # Kibana (Visualization)
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Logstash (Log Processing)
  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    ports:
      - "5044:5044"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    environment:
      - xpack.monitoring.enabled=false
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Filebeat (Log Shipper)
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    command: filebeat -e -strict.perms=false
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - logstash
    networks:
      - elk

  # WSO2 Integrator
  integrator:
    image: eclipse-temurin:21-jre
    working_dir: /app
    command: sh -c "java -jar target/bin/*.jar"
    ports:
      - "8090:9090"
      - "9797:9797"
    environment:
      - BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSENABLED=true
      - BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSREPORTER=prometheus
      - BAL_CONFIG_VAR_BALLERINAX_PROMETHEUS_PORT=9797
      - BAL_CONFIG_VAR_BALLERINA_LOG_LEVEL=INFO
      - BAL_CONFIG_VAR_BALLERINA_LOG_FORMAT=json
    volumes:
      - ./integration:/app
    networks:
      - elk

  # Prometheus (Optional, for metrics)
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - elk

  # Grafana (Optional, for metrics visualization)
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    depends_on:
      - prometheus
    networks:
      - elk

volumes:
  elasticsearch-data:
  prometheus-data:
  grafana-data:

networks:
  elk:
    driver: bridge

Configuration Files

Integration configuration

Enable observability in the integration's Ballerina.toml:

[build-options]
observabilityIncluded = true

Import the Prometheus extension in one of the integration's .bal files:

import ballerinax/prometheus as _;

Filebeat Configuration (`filebeat.yml`)

Filebeat discovers the integrator container through the Docker socket and ships only that container's logs:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.labels.com.docker.compose.service: integrator
          config:
            - type: filestream
              id: integrator-${data.docker.container.id}
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              parsers:
                - container: ~
                - ndjson:
                    target: ""
                    overwrite_keys: true
                    add_error_key: true
              fields:
                service: wso2-integrator
                environment: production
              fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info

Logstash Configuration (`logstash.conf`)

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "json" {
    json {
      source => "message"
    }
  }

  # Parse timestamp
  if [time] {
    date {
      match => ["time", "ISO8601"]
      target => "@timestamp"
    }
  }

  # Add service metadata
  mutate {
    add_field => {
      "log_source" => "ballerina-integration"
      "processing_timestamp" => "%{@timestamp}"
    }

    # Remove empty fields
    remove_field => ["[input][type]"]
  }

  # Extract orderId if present
  if [orderId] {
    mutate {
      add_field => { "trace_id" => "%{orderId}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "ballerina-integrations-%{+yyyy.MM.dd}"
  }

  # Optional: Forward to stdout for debugging
  # stdout {
  #   codec => rubydebug
  # }
}

Prometheus Configuration (`prometheus.yml`)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "wso2-integrator"
    static_configs:
      - targets: ["integrator:9797"]

Grafana Datasources (`grafana-datasources.yml`)

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Running the Stack

# Build the integration
cd integration
bal build
cd ..

# Start all services
docker compose up -d

# Verify services are running
docker compose ps

# View logs
docker compose logs -f

Accessing the Services

Kibana: http://localhost:5601
Elasticsearch: http://localhost:9200
Grafana (metrics): http://localhost:3000
Prometheus: http://localhost:9090

Setting up Kibana data view

Open Kibana at http://localhost:5601
Go to Stack Management > Data Views
Click Create data view
Index pattern: ballerina-integrations-*
Timestamp field: @timestamp
Click Save data view to Kibana

Creating Kibana Visualizations

1. Log Volume Over Time

Go to Visualize > Create Visualization
Select Area chart type
Elasticsearch index: ballerina-integrations-*
X-axis: Date Histogram on @timestamp
Y-axis: Count
Add filter: level: "ERROR" (optional)

2. Error Rate Table

Create Data Table visualization
Elasticsearch index: ballerina-integrations-*
Bucket aggregation: Terms on level.keyword
Metric: Count
Filter: level: "ERROR"

3. Service Breakdown Pie Chart

Create Pie visualization
Elasticsearch index: ballerina-integrations-*
Bucket: Terms on module.keyword
Metric: Count

4. Custom Kibana Dashboard

Go to Dashboards > Create Dashboard
Add your visualizations
Search by orderId: orderId: "ORD-12345"
Search by log level: level: "ERROR" OR level: "WARN"
Search by service: module: "myorg/order_service"

Useful KQL Queries

Query	Purpose
`level: "ERROR"`	Find all error logs
`service: "wso2-integrator" AND orderId: "ORD-123"`	Trace a specific order
`level: "ERROR" AND NOT message: "timeout"`	Errors excluding timeouts
`message: "Payment failed"`	Search error messages
`orderId: *`	Logs with orderId field
`level: "WARN" OR level: "ERROR"`	Warnings and errors

Advanced: Logstash Parsing Examples

These patterns extract values written into the log message text itself, not values passed as separate key-value pairs.

Extract Order Information

filter {
  if [message] =~ /Order received/ {
    grok {
      match => {
        "message" => "Order received.*orderId=%{DATA:orderId}.*customerId=%{DATA:customerId}"
      }
    }
  }
}

Extract API Response Times

filter {
  if [message] =~ /response_time/ {
    grok {
      match => {
        "message" => "API call.*response_time=%{NUMBER:response_time:float}ms"
      }
    }
    mutate {
      convert => { "response_time" => "float" }
    }
  }
}

Performance Tuning

Elasticsearch JVM Heap

# Edit docker-compose.yml
environment:
  - "ES_JAVA_OPTS=-Xms2g -Xmx2g"

Index Retention Policy

# Delete an old index by name (Elasticsearch rejects wildcard deletes by default)
curl -X DELETE "localhost:9200/ballerina-integrations-2024.01.01"

Logstash Pipeline Config

# Optimize batch settings (logstash service in docker-compose.yml)
environment:
  - PIPELINE_BATCH_SIZE=500
  - PIPELINE_BATCH_DELAY=50

Troubleshooting

No logs appearing in Kibana:

# Check Filebeat is running
docker compose logs filebeat

# Check Logstash is receiving beats
docker compose logs logstash | grep "beats"

# Check Elasticsearch has data
curl http://localhost:9200/_cat/indices

Index pattern not found:

# Verify index exists
curl http://localhost:9200/_cat/indices?v

# Wait for data to arrive (can take 30 seconds)
docker compose logs integrator | tail -20

High disk usage:

Reduce log retention: index => "ballerina-integrations-%{+yyyy.MM}"
Delete old indices regularly
Increase Elasticsearch volume size

Cleanup

# Stop all services
docker compose down

# Remove all data
docker compose down -v

What's Next

Elastic Stack (ELK) — The focused integration guide for this platform
OpenSearch Setup – Open-source alternative
Logging Overview – Learn more about logging
Metrics Overview – Add metrics collection

Architecture​

Prerequisites​

Docker Compose Setup​

Configuration Files​

Integration configuration​

Filebeat Configuration (filebeat.yml)​

Logstash Configuration (logstash.conf)​

Prometheus Configuration (prometheus.yml)​

Grafana Datasources (grafana-datasources.yml)​

Running the Stack​

Accessing the Services​

Setting up Kibana data view​

Creating Kibana Visualizations​

1. Log Volume Over Time​

2. Error Rate Table​

3. Service Breakdown Pie Chart​

4. Custom Kibana Dashboard​

Useful KQL Queries​

Advanced: Logstash Parsing Examples​

Extract Order Information​

Extract API Response Times​

Performance Tuning​

Elasticsearch JVM Heap​

Index Retention Policy​

Logstash Pipeline Config​

Troubleshooting​

Cleanup​

What's Next​