Observability

When you run an agent, you want to see exactly what it did. Which tools it picked, what it sent to the LLM, what came back, and how long each step took. WSO2 Agent Builder ships with a dev-time trace server, a built-in tracing backend that runs alongside the editor and collects spans as your agentic integration executes. With tracing on, the LLM's tool choices and interpretations of each result are visible in the spans, so you can debug agent behavior without adding print statements.

Enable tracing

Tracing is off by default, and it applies to the whole agentic integration.

From the agent canvas

The agent canvas has a Tracing toggle in the top-right corner. When it's off, the toggle reads Tracing: Off. The same toggle is also on the Chat Agent Service view of an Agent Chat trigger.

Click the toggle to turn it on. The label changes to Tracing: On, and a Dev-time tracing enabled. notification appears in the bottom-right.

Click the toggle again at any time to turn tracing back off.

From the command palette

You can also turn tracing on and off from the command palette, for example when the agent canvas isn't open.

1. Open the command palette with Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux).
2. Run Ballerina: Enable Tracing.

To stop tracing, run Ballerina: Disable Tracing the same way.

View traces

There are two ways to look at traces from a running agent: the View Trace link inside the Agent Chat panel, and the Traces panel at the bottom of the editor.

From the chat window

When you chat with an agent through an Agent Chat trigger (see Try it and run), every agent response carries an Execution Steps list and a View Trace link, and the chat header has a Session Traces button.

View Trace

Click View Trace on any reply to open the detailed trace for that single message in the Trace Logs view.

The viewer has three regions: a span list on the left, span detail in the center, and the original chat on the right. The chat stays visible, so you can correlate spans with the message you're investigating without losing context. See The trace viewer below for what each region offers.

To jump straight to a specific span without scanning the list, expand the Execution Steps dropdown above an agent reply. It lists every Chat and Execute Tool span for that turn with its duration. Click any step to load it in the center pane.

Session Traces

Click Session Traces in the chat header to open a table of every trace produced in the current chat session.

Each row is one chat turn, with its timestamp, trace ID, user input, the agent's reply, latency, and token counts. Click a trace ID to open its detail view (the same view as View Trace). Use the search bar to filter by message text or trace ID. Use Export to save the session for sharing or later analysis (see Export traces).

From the Traces panel

For agents without a chat window, or when you want a flat list of every trace the agentic integration has produced, use the Traces panel at the bottom of the editor. Open the panel with Cmd+J (macOS) or Ctrl+J (Windows/Linux) and switch to the Traces tab, or run Traces: Focus on Tracing View from the command palette. As the agentic integration runs, traces stream in.

Click a trace to open it in the Trace Logs view. Its span tree shows the top-level service handler (for example, Server post /chat), the agent invocation, the LLM calls it makes, each tool invocation, and any HTTP calls.

By default the panel shows every trace the agentic integration produces, including traces for unrelated services. Click Filter Agent Traces (the agent icon) in the panel's toolbar to show agent traces only. A Showing agent traces only note appears at the top of the list while the filter is active. The toolbar also has Clear Traces, Disable Tracing, and Collapse All.

The trace viewer

The trace viewer is the Trace Logs view that opens when you click View Trace, a trace ID, or a trace in the Traces panel. It's built specifically for understanding agent runs, so most of what you need is in this one view.

Span list

The left sidebar lists every span in the run, in the order it executed and indented to show parent-child relationships. The span types you'll see most often are:

• Server. The service resource that received the request, for example post /chat.
• Invoke Agent. The top-level call to the agent. One per chat turn.
• Chat. A call to the underlying LLM (for example, Chat gpt-4o-mini). An agent typically makes several Chat spans per turn. One for the initial decision, one after each round of tool results, and one for the final answer.
• Execute Tool. A tool the agent picked (for example, Execute Tool getOrderStatus).
• Client. Outbound HTTP calls, such as the call to the model provider or calls made from inside a tool. Captured automatically.

Each row shows its duration and token usage where applicable, so the slowest or most expensive step in a turn is visible at a glance. Use the Filter spans... field above the list to narrow it down on long traces.

Click Timeline at the top of the sidebar to switch from the list view to a Gantt-style chart. Each span becomes a horizontal bar laid out across a time axis, so you can see when each step started and how long it took relative to the rest of the run. This is the fastest way to spot a long Chat span sitting between two quick tool calls, or to confirm that two operations overlapped. Click any bar to load that span's detail in the center pane, just like in the list view.

Quick info pills

Each span shows a row of quick info pills under its title, summarizing the metadata you care about most. The exact set depends on the span type.

For a Chat span, you typically see:

• Latency. Wall-clock time the LLM call took.
• Input Tokens and Output Tokens. Exact token counts for that call.
• Temperature. Sampling temperature used for the request.
• Provider and Model. Which LLM the call went to.
• Start Time and End Time. Useful when correlating with logs.

For an Invoke Agent span, the pills show aggregate totals across the whole turn (overall latency, total input tokens, total output tokens), so the parent span tells you the cost of the turn at a glance.

Input and Output sections

Below the pills, every span has an Input section and an Output section, each collapsible. They contain the exact data that flowed in and out of that step.

What you see depends on the span:

• Invoke Agent. Input lists the System Instructions (Role and Instructions) and the User message. Output is the agent's final reply.
• Chat. Input lists the Messages sent to the LLM (the full chat thread, including system, user, assistant, and tool messages) plus the Available Tools definitions. Output is the model's response, including any tool calls it requested.
• Execute Tool. Input is the arguments the agent passed to the tool. Output is the value the tool returned.

Every span also has an Advanced Details section, collapsed by default. It lists the span's Technical IDs (span ID, trace ID, parent span ID, and start and end times) and its Other Attributes, such as the OpenTelemetry gen_ai.* attributes and the entry point the request came through.

For an Execute Tool span, Input shows the named arguments the agent supplied and Output shows the returned value. The span header also carries a Tool Description pill so you can see what the agent thought the tool did.

Formatted and raw views

Each section has a Formatted and JSON toggle (labeled Raw in some sections; both mean the underlying payload, untransformed).

Formatted renders the payload as an expandable tree, with field names highlighted and nested objects collapsible. Use this view when reading a prompt or scanning tool calls.

JSON shows the underlying JSON exactly as the agent saw it. Use this view when copying a payload into a bug report or comparing two runs character-for-character.

The toggle is per-section, so within a single span, you can keep Messages formatted while flipping Available Tools to JSON.

Search

The detail pane has a Search all sections... field that filters within the current span's payload across both Input and Output. This is useful when a Chat span's input is several thousand tokens long and you want to find a specific tool name, parameter, or piece of text.

Export traces

Traces are useful beyond the editor, in bug reports, regression baselines, or evaluation runs. The trace viewer can export in two formats:

• JSON. The full trace as structured data. Best for archiving, sharing in a bug report, or comparing two runs.
• Evalset. The trace converted into an evaluation set you can run later through the testing framework. See Evaluations for the full workflow.

You can export a single trace or every trace in a chat session.

Export a single trace

In the trace viewer, click the Export (download) icon at the top of the Trace sidebar. Choose Export as JSON or Export as Evalset.

The exported file contains the spans for that one chat turn only.

Export a session

In the Session Traces view, click Export in the top-right and choose Export as JSON or Export as Evalset.

The exported file contains every trace in the current session, in the order they ran. Use this when you want to capture an entire conversation, for example to reproduce an issue end-to-end or to seed a regression suite.

Connect to an external trace provider

The trace viewer described above is wired to the built-in editor trace provider. To send traces to your existing observability stack instead (Jaeger, Zipkin, or any OpenTelemetry-compatible collector), swap the trace provider in your project files.

The example below uses Jaeger. For other providers, follow the same pattern with the matching ballerinax extension.

Three changes are required:

1. In Ballerina.toml, include observability in the build:

   [build-options]
   observabilityIncluded = true

2. In any .bal file in your agentic integration (for example, trace_enabled.bal), import the provider extension. The as _ prefix is required because the module is imported only for its side effects:

   import ballerinax/jaeger as _;

3. In Config.toml, enable tracing and select the provider:

   [ballerina.observe]
   tracingEnabled = true
   tracingProvider = "jaeger"

When you run the agentic integration, spans flow to the configured collector instead of the editor trace viewer. To point at a non-local collector or tune sampling, add a [ballerinax.jaeger] section to Config.toml with the provider-specific configuration keys.

Export over OTLP

To send traces to an OpenTelemetry Collector, or to any backend that accepts OTLP, use the ballerina/otel module in place of the Jaeger extension. Keep observabilityIncluded = true in Ballerina.toml, then:

1. Import the module:

   import ballerina/otel as _;

2. In Config.toml, select the otel provider and set the endpoint:

   [ballerina.observe]
   tracingEnabled = true
   tracingProvider = "otel"
   
   [ballerina.otel]
   tracesEndpoint = "http://127.0.0.1:4317"
   tracesProtocol = "grpc"
   
   [ballerina.otel.tracesResourceAttributes]
   "service.name" = "my-agent"

ballerina/otel sends requests without signing them. A backend that requires signed or authenticated requests, such as AWS X-Ray, needs an OpenTelemetry Collector in between to add the credentials.

Agent spans follow the OpenTelemetry GenAI semantic conventions, which most AI observability backends understand. A few details matter when a backend expects specific attribute names:

Span	Attribute	Holds
invoke_agent (one per agent run)	gen_ai.input.messages, gen_ai.output.messages	The prompt and the agent's answer
invoke_agent	gen_ai.conversation.id	The sessionId passed to agent.run()
execute_tool (one per tool call)	gen_ai.tool.arguments, gen_ai.tool.output	The tool's arguments and its result
LLM call spans	gen_ai.usage.input_tokens, gen_ai.usage.output_tokens	Token counts, recorded as strings
All spans	Instrumentation scope name otel	The instrumentation that produced the span; some backends select spans by it

If your backend expects other names, for example session.id for the session or integer token counts, rename or convert the attributes in the collector rather than in the agent.

To switch back to the dev-time viewer from either provider, change the import to ballerinax/idetraceprovider as _; and set tracingProvider = "idetraceprovider" in Config.toml.

What's next

• Evaluations — turn observed behavior into automated regression checks.
• Tools — clear tool descriptions are the biggest lever on what shows up in traces.
• Memory — every turn's memory contents appear in the LLM call spans.