Version: 5.1

OpenSearch

With OpenSearch, Fluent Bit collects both application and metrics logs, stores them in separate OpenSearch indices, and ICP queries those indices to display logs and metrics. Follow the steps below to deploy and configure this stack.

tip

This guide covers default profile runtimes. To set up observability for a WSO2 Integrator: MI runtime connected to ICP, see Adding observability for ICP in the MI documentation.

Prerequisites

ICP installed and running. See Install ICP.
Integration connected to ICP with heartbeats working. See Connect an integration to ICP.

Architecture

The default profile runtime writes structured logs to two files (app.log and metrics.log) and sends periodic heartbeats to the ICP server.
Fluent Bit tails both log files and ships them to separate OpenSearch indices over HTTP(S).
When a user opens the Logs or Metrics page, the ICP console sends a GraphQL or REST request to the ICP server, which queries OpenSearch and returns the results.

1. Deploy OpenSearch

Any single-node or clustered OpenSearch deployment works. ICP requires HTTP(S) access to the OpenSearch REST API. Keep a note of the OpenSearch host, port, and credentials. You will need them throughout the following steps.

If you already have an OpenSearch instance running, skip the installation steps and continue with Configure ICP to connect to OpenSearch, then create the index templates in step 2.

Download and extract the OpenSearch distribution, then follow the steps below.

macOS prerequisite

For a tarball installation on macOS, install a Java version supported by your chosen OpenSearch release and set JAVA_HOME to that installation before running the commands below. See the OpenSearch tarball installation guide.

Run the security installer

Set your admin password and run the demo installer. It generates self-signed TLS certificates and initializes the security index.

Linux / macOS
Windows

export OPENSEARCH_INITIAL_ADMIN_PASSWORD="<your-opensearch-password>"
export OPENSEARCH_HOME="/path/to/opensearch-<version>"

cd $OPENSEARCH_HOME/plugins/opensearch-security/tools
chmod +x install_demo_configuration.sh
./install_demo_configuration.sh -y

$env:OPENSEARCH_INITIAL_ADMIN_PASSWORD="<your-opensearch-password>"
$env:OPENSEARCH_HOME="C:\opensearch\opensearch-<version>"

cd $env:OPENSEARCH_HOME\plugins\opensearch-security\tools
cmd /c install_demo_configuration.bat -y

Configure opensearch.yml

Append the following to config/opensearch.yml. The first setting prevents OpenSearch from waiting for other cluster nodes, and the second allows the security index to initialize automatically on first start:

discovery.type: single-node
plugins.security.allow_default_init_securityindex: true

Start OpenSearch

Linux / macOS
Windows

cd $OPENSEARCH_HOME/bin
./opensearch

cd $env:OPENSEARCH_HOME\bin
.\opensearch.bat

Verify OpenSearch is running

The curl examples in this guide verify HTTPS certificates by default. If you use the self-signed certificate generated by the demo installer, add -k. If your certificate is signed by a private CA that is not in the system trust store, add --cacert <path-to-ca-certificate> instead.

curl -s -u 'admin:<your-opensearch-password>' 'https://<opensearch-host>:<opensearch-port>'

Replace <opensearch-host> and <opensearch-port> with the configured OpenSearch host and port. Use localhost only for a node on the same machine. A JSON response with "cluster_name" confirms the node is up.

warning

The demo configuration is for evaluation only. In production, use properly signed certificates and strong credentials. See the OpenSearch security documentation for details.

Configure ICP to connect to OpenSearch

Once OpenSearch is running, point the ICP server at it by adding the following keys before the first [section] header in conf/deployment.toml in your ICP installation directory:

opensearchUrl      = "https://<opensearch-host>:<opensearch-port>"
opensearchUsername = "admin"
opensearchPassword = "<your-opensearch-password>"

Replace <opensearch-host> and <opensearch-port> with the host and port configured for OpenSearch. Use localhost only when OpenSearch runs on the same host as ICP. If OpenSearch runs without TLS, use http:// instead.

Once the configuration is saved, start the ICP server:

Linux / macOS
Windows

./bin/icp.sh

.\bin\icp.bat

If ICP is already running, restart it for the OpenSearch connection settings to take effect.

2. Create index templates

Index templates tell OpenSearch how to map fields before any data arrives. Apply both templates once per cluster. They cover all indices created by Fluent Bit in the steps ahead.

Application logs template

Maps the time, level, message, and icp_runtimeId fields for the ballerina-application-logs-* index pattern:

Linux / macOS
Windows

curl -X PUT 'https://<opensearch-host>:<opensearch-port>/_index_template/wso2_integration_application_log_template' \
  -u 'admin:<your-opensearch-password>' \
  -H 'Content-Type: application/json' \
  -d '{
    "index_patterns": ["ballerina-application-logs-*"],
    "template": {
      "mappings": {
        "properties": {
          "time":           { "type": "date", "format": "yyyy-MM-dd'\''T'\''HH:mm:ss.SSS'\''Z'\''||strict_date_optional_time||epoch_millis" },
          "level":          { "type": "keyword" },
          "message":        { "type": "text" },
          "icp_runtimeId":  { "type": "keyword" }
        }
      }
    }
  }'

First, save the following JSON to a file named app-log-template.json:

{
  "index_patterns": ["ballerina-application-logs-*"],
  "template": {
    "mappings": {
      "properties": {
        "time": { "type": "date", "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'||strict_date_optional_time||epoch_millis" },
        "level": { "type": "keyword" },
        "message": { "type": "text" },
        "icp_runtimeId": { "type": "keyword" }
      }
    }
  }
}

Then run in Command Prompt:

curl -X PUT "https://<opensearch-host>:<opensearch-port>/_index_template/wso2_integration_application_log_template" -u "admin:<your-opensearch-password>" -H "Content-Type: application/json" -d @app-log-template.json

A {"acknowledged":true} response confirms the template was created.

Metrics logs template

Maps the time and message fields plus numeric types for the latency fields used by the ICP Metrics page:

Linux / macOS
Windows

curl -X PUT 'https://<opensearch-host>:<opensearch-port>/_index_template/wso2_integration_metrics_log_template' \
  -u 'admin:<your-opensearch-password>' \
  -H 'Content-Type: application/json' \
  -d '{
    "index_patterns": ["ballerina-metrics-logs-*"],
    "template": {
      "mappings": {
        "properties": {
          "time":                   { "type": "date", "format": "yyyy-MM-dd'\''T'\''HH:mm:ss.SSS'\''Z'\''||strict_date_optional_time||epoch_millis" },
          "message":                { "type": "text" },
          "response_time_seconds":  { "type": "float" },
          "response_time":          { "type": "long" }
        }
      }
    }
  }'

First, save the following JSON to a file named metrics-log-template.json:

{
  "index_patterns": ["ballerina-metrics-logs-*"],
  "template": {
    "mappings": {
      "properties": {
        "time": { "type": "date", "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'||strict_date_optional_time||epoch_millis" },
        "message": { "type": "text" },
        "response_time_seconds": { "type": "float" },
        "response_time": { "type": "long" }
      }
    }
  }
}

Then run in Command Prompt:

curl -X PUT "https://<opensearch-host>:<opensearch-port>/_index_template/wso2_integration_metrics_log_template" -u "admin:<your-opensearch-password>" -H "Content-Type: application/json" -d @metrics-log-template.json

A {"acknowledged":true} response confirms the template was created.

3. Enable observability in the integration

These changes are additive and build on the base configuration from Connect an integration to ICP. You need to update three files.

Ballerina.toml

Add observabilityIncluded alongside the existing remoteManagement build option:

[build-options]
remoteManagement = true
observabilityIncluded = true

Without observabilityIncluded = true, the ballerinax/metrics.logs module cannot emit per-request metrics.

main.bal

Add the metrics logger import alongside the existing bridge import:

import ballerinax/metrics.logs as _;
import wso2/icp.runtime.bridge as _;

Both are blank imports (as _). They activate automatically at startup.

info

Ensure to keep the import order as shown.

Config.toml

Add logging and metrics settings alongside the existing [wso2.icp.runtime.bridge] section:

[ballerina.observe]
metricsLogsEnabled = true

[ballerina.log]
format = "logfmt"

[[ballerina.log.destinations]]
path = "./logs/app.log"

[ballerinax.metrics.logs]
logFilePath = "./logs/metrics.log"

This produces two separate log files that Fluent Bit will tail in step 4:

File	Content	OpenSearch index
`logs/app.log`	Application logs (startup, errors, user log statements)	`ballerina-application-logs-*`
`logs/metrics.log`	Per-request metrics (response times, status codes, endpoints)	`ballerina-metrics-logs-*`

Setting	Purpose
`metricsLogsEnabled = true`	Enables the Ballerina runtime to emit per-request metrics
`format = "logfmt"`	Structured log format parsed by `bal_logfmt_filter_parser` after Lua preprocessing
`path`	Application log destination
`logFilePath`	Metrics log destination (written by `ballerinax/metrics.logs`)

warning

The log file paths must match the Fluent Bit Path values configured in step 4. Adjust both sides if you change the directory layout.

Start the application

Run the integration from your project directory:

bal run

On startup, the runtime creates the logs/ directory and begins writing to app.log and metrics.log. These files are what Fluent Bit will tail in step 4.

4. Configure Fluent Bit

Fluent Bit tails the default profile log files and ships them to OpenSearch.

Install Fluent Bit

If you have not installed Fluent Bit yet, download it from the Fluent Bit installation page. Once installed, verify it is available:

fluent-bit --version

Set up the working directory

Run the following commands to create the directory structure and placeholder files:

Linux / macOS
Windows

mkdir -p fluent-bit/scripts
touch fluent-bit/fluent-bit.conf fluent-bit/parsers.conf fluent-bit/scripts/scripts.lua

New-Item -ItemType Directory -Force -Path fluent-bit\scripts
New-Item -ItemType File -Force -Path fluent-bit\fluent-bit.conf, fluent-bit\parsers.conf, fluent-bit\scripts\scripts.lua

This produces the following layout:

fluent-bit/
├── fluent-bit.conf
├── parsers.conf
└── scripts/
    └── scripts.lua

parsers.conf

Defines how Fluent Bit parses Ballerina's logfmt output after Lua preprocessing. Both inputs use this parser through a parser filter. The time field is kept as a regular field for ICP's time-range queries. Copy the following into parsers.conf:

[PARSER]
    Name        bal_logfmt_filter_parser
    Format      logfmt
    Types       response_time_seconds:float

scripts.lua

Preprocesses raw log lines to preserve JSON objects and arrays and normalize icp.runtimeId to icp_runtimeId before logfmt parsing. It then enriches each log record with fields the ICP Metrics page needs, including response_time in milliseconds, status, app, and integration. It also generates a hash-based doc_id for deduplication in OpenSearch.

Copy the content from scripts.lua and paste it into the scripts/scripts.lua file created earlier.

Function	Purpose
`preprocess_bal_log`	Preserves structured values and normalizes the runtime ID before logfmt parsing
`extract_app_from_path`	Derives `app_name` from the log file path
`enrich_bal_logs`	Adds `product` and `app_module` fields
`construct_bal_app_name`	Builds the `app` and `deployment` fields
`extract_bal_metrics_data`	Parses metrics-specific fields (response time in ms, status, method, URL)
`generate_document_id`	Creates a hash-based `doc_id` for deduplication

warning

The Lua enrichment is required for the ICP Metrics page to display data. Without extract_bal_metrics_data, the ICP server cannot categorize inbound vs. outbound metrics, and the Metrics page shows "No metrics data" even when the underlying OpenSearch index contains records.

fluent-bit.conf

The main pipeline configuration. It reads app.log and metrics.log, enriches each record through the Lua scripts, and writes to separate OpenSearch indices:

Input path	Tag	Output index prefix
`<default-profile-logs>/app.log`	`ballerina_app_logs`	`ballerina-application-logs-`
`<default-profile-logs>/metrics.log`	`ballerina_metrics`	`ballerina-metrics-logs-`

Before using this config, replace the placeholders in both inputs and outputs:

<default-profile-logs>: absolute path to your default profile application's logs/ directory. Use forward slashes on all platforms.
<opensearch-host>: the OpenSearch hostname reachable from Fluent Bit. Use localhost only when OpenSearch is on the same host.
<opensearch-port>: the OpenSearch REST API port, usually 9200.
<opensearch-username> and <your-opensearch-password>: the credentials Fluent Bit uses to write to OpenSearch.

The example uses HTTPS with the demo certificate. For plain HTTP, set tls Off in both outputs. For production HTTPS, enable certificate verification and configure the trusted CA as needed.

[SERVICE]
    Flush        1
    Parsers_File parsers.conf
    Log_Level    info

# ── App logs ──
[INPUT]
    Name         tail
    Path         <default-profile-logs>/app.log
    Tag          ballerina_app_logs
    Read_from_Head On
    Path_Key     log_file_path

# ── Metrics logs ──
[INPUT]
    Name         tail
    Path         <default-profile-logs>/metrics.log
    Tag          ballerina_metrics
    Read_from_Head On
    Path_Key     log_file_path

# ── Preprocess raw lines before parsing ──
[FILTER]
    Name    lua
    Match   ballerina_*
    Script  scripts/scripts.lua
    Call    preprocess_bal_log

# ── Parse logfmt and preserve the preprocessed fields ──
[FILTER]
    Name         parser
    Match        ballerina_*
    Key_Name     log
    Parser       bal_logfmt_filter_parser
    Reserve_Data On

# ── Enrich app logs ──
[FILTER]
    Name    lua
    Match   ballerina_app_logs
    Script  scripts/scripts.lua
    Call    extract_app_from_path

[FILTER]
    Name    lua
    Match   ballerina_app_logs
    Script  scripts/scripts.lua
    Call    enrich_bal_logs

[FILTER]
    Name    lua
    Match   ballerina_app_logs
    Script  scripts/scripts.lua
    Call    construct_bal_app_name

# ── Enrich metrics logs ──
[FILTER]
    Name    lua
    Match   ballerina_metrics
    Script  scripts/scripts.lua
    Call    extract_app_from_path

[FILTER]
    Name    lua
    Match   ballerina_metrics
    Script  scripts/scripts.lua
    Call    enrich_bal_logs

[FILTER]
    Name    lua
    Match   ballerina_metrics
    Script  scripts/scripts.lua
    Call    construct_bal_app_name

[FILTER]
    Name    lua
    Match   ballerina_metrics
    Script  scripts/scripts.lua
    Call    extract_bal_metrics_data

# ── Document IDs (deduplication) ──
[FILTER]
    Name    lua
    Match   ballerina_app_logs
    Script  scripts/scripts.lua
    Call    generate_document_id
    time_as_table true

[FILTER]
    Name    lua
    Match   ballerina_metrics
    Script  scripts/scripts.lua
    Call    generate_document_id
    time_as_table true

# ── Outputs ──
[OUTPUT]
    Name            opensearch
    Match           ballerina_app_logs
    Host            <opensearch-host>
    Port            <opensearch-port>
    Logstash_Format On
    Logstash_Prefix ballerina-application-logs
    Replace_Dots    On
    Suppress_Type_Name On
    Id_Key          doc_id
    Write_Operation index
    tls             On
    tls.verify      Off
    HTTP_User       <opensearch-username>
    HTTP_Passwd     <your-opensearch-password>

[OUTPUT]
    Name            opensearch
    Match           ballerina_metrics
    Host            <opensearch-host>
    Port            <opensearch-port>
    Logstash_Format On
    Logstash_Prefix ballerina-metrics-logs
    Replace_Dots    On
    Suppress_Type_Name On
    Id_Key          doc_id
    Write_Operation index
    tls             On
    tls.verify      Off
    HTTP_User       <opensearch-username>
    HTTP_Passwd     <your-opensearch-password>

Setting	Notes
`tls On` / `tls.verify Off`	Required when using the demo setup from step 1 (self-signed certificate). Set `tls Off` for plain HTTP.
`HTTP_User` / `HTTP_Passwd`	OpenSearch credentials set during setup.
`Id_Key doc_id` / `Write_Operation index`	Uses the generated document ID to replace an existing document with the same ID in the target index instead of creating a duplicate.
`Replace_Dots On`	Converts dotted field names (e.g. `src.module`, `http.method`) to the underscore names expected by ICP's OpenSearch queries.

Start Fluent Bit

Run Fluent Bit from the fluent-bit/ directory created above so the relative parser and Lua script paths resolve correctly:

Linux / macOS
Windows

cd /path/to/fluent-bit
fluent-bit -c fluent-bit.conf

Set-Location "C:\path\to\fluent-bit"
fluent-bit -c fluent-bit.conf

Fluent Bit will begin tailing the log files immediately. Check the console output for any connection errors to OpenSearch.

Verification

Check OpenSearch indices

After the default profile runtime has been running for a minute or two:

curl -s -u '<opensearch-username>:<your-opensearch-password>' 'https://<opensearch-host>:<opensearch-port>/_cat/indices/ballerina-*?v'

You should see:

ballerina-application-logs-2026.04.30
ballerina-metrics-logs-2026.04.30

Use the host, port, and credentials for your OpenSearch deployment. For plain HTTP OpenSearch (no TLS), use http://. On Windows, run this command in PowerShell using curl.exe in place of curl.

Check ICP console

Log into the ICP console at https://<icp-host>:9446.
Navigate to Projects > <project> > Integrations > <integration>.
The integration overview shows the service endpoints and environment cards with runtime status.
Click Logs in the sidebar. You should see runtime log entries with timestamps, log levels, and messages. Use the environment, level, and time range filters to narrow results.
Click Metrics in the sidebar. You should see:
- Summary cards: Total Requests, Error Count, Error Percentage, 95th Percentile latency
- Requests Per Minute chart (success vs. failed)
- Request Latency chart (average, P50, P95, P99)
- Most Used APIs table showing each endpoint with request count, error count, and average response time

tip

Metrics are generated per inbound HTTP request. If the Metrics page shows "No metrics data", send some traffic to your integration first:

curl 'http://localhost:9090/<your-endpoint>'

Troubleshooting

Symptom	Cause	Fix
Metrics page shows "No metrics data"	No inbound HTTP requests have been made	Metrics are generated per request. Send traffic to the integration first.
Metrics page shows "No metrics data"	`metricsLogsEnabled` not set	Add `metricsLogsEnabled = true` to `[ballerina.observe]` in `Config.toml`
Metrics page shows "No metrics data"	Metrics log file not configured	Set `logFilePath` in `[ballerinax.metrics.logs]`
Metrics page shows "No metrics data"	Lua enrichment scripts missing from Fluent Bit config	Add the Lua `[FILTER]` blocks (especially `extract_bal_metrics_data`). See step 4.
Logs page shows "To enable the logs view you need to setup observability with either Moesif or OpenSearch" instead of logs	ICP Server can't reach OpenSearch	Verify `opensearchUrl` in ICP Server's `deployment.toml`
OpenSearch rejects documents with "total fields [1000] exceeded"	Deeply nested JSON in log messages	Increase limit: `curl -X PUT 'https://<opensearch-host>:<opensearch-port>/ballerina-application-logs-*/_settings' -u 'admin:<your-opensearch-password>' -H 'Content-Type: application/json' -d '{"index.mapping.total_fields.limit": 2000}'` or add to the index template

Index lifecycle

Indices are created daily with a date suffix (e.g. ballerina-metrics-logs-2026.04.28). To manage disk usage:

Use OpenSearch Index State Management (ISM) policies to automatically delete or roll over old indices.
A typical retention policy keeps 30 days of logs and 90 days of metrics.

Security notes

In production, enable TLS on OpenSearch and set tls.verify On in Fluent Bit.
Use dedicated OpenSearch credentials for Fluent Bit (write-only) and ICP server (read-only).
The ICP server generates short-lived JWTs (2 min) for internal communication between its observability service and its OpenSearch adapter. No user configuration is needed.

What's next

Manage integrations — create integrations and navigate the integration overview in the ICP console
Manage runtimes — monitor runtime health and status alongside observability data
Access control — control who can view logs and metrics in the ICP console

Architecture​

1. Deploy OpenSearch​

Run the security installer​

Configure opensearch.yml​

Start OpenSearch​

Verify OpenSearch is running​

Configure ICP to connect to OpenSearch​

2. Create index templates​

Application logs template​

Metrics logs template​

3. Enable observability in the integration​

Ballerina.toml​

main.bal​

Config.toml​

Start the application​

4. Configure Fluent Bit​

Install Fluent Bit​

Set up the working directory​

parsers.conf​

scripts.lua​

fluent-bit.conf​

Start Fluent Bit​

Verification​

Check OpenSearch indices​

Check ICP console​

Troubleshooting​

Index lifecycle​

Security notes​

What's next​

Architecture

1. Deploy OpenSearch

Run the security installer

Configure opensearch.yml

Start OpenSearch

Verify OpenSearch is running

Configure ICP to connect to OpenSearch

2. Create index templates

Application logs template

Metrics logs template

3. Enable observability in the integration

Ballerina.toml

main.bal

Config.toml

Start the application

4. Configure Fluent Bit

Install Fluent Bit

Set up the working directory

parsers.conf

scripts.lua

fluent-bit.conf

Start Fluent Bit

Verification

Check OpenSearch indices

Check ICP console

Troubleshooting

Index lifecycle

Security notes

What's next