Evaluations
Evaluations are how you keep an AI agent's quality from regressing. Unlike unit tests with deterministic outputs, agent behaviour has to be judged across several dimensions: correctness, tool selection, groundedness, safety, and tone. Evaluations in WSO2 Integrator give you a structured way to measure those dimensions, see why a run regressed, and watch the trend across builds.
The feature is built around three stages.
| Stage | What it is | Page |
|---|---|---|
| Evalsets | A golden dataset of conversation traces, captured from real chats with your agent and refined in the Evalset Viewer. | Create evalsets |
| Evaluations | Evaluation functions that score agent behaviour, created from a prebuilt template or built in the visual designer, with or without an evalset. | Templates, Custom |
| Runs and reports | Run an evaluation on the current agent build and review the Evaluation Report and the Evaluation History trend across runs. | Run evaluations |
How the stages fit together
- Chat with your agent and export the session traces into an evalset. Use the editor to edit messages, reorder or add turns, and update tool calls.
- Create an evaluation from the Test Explorer. Start from a prebuilt template for common checks such as tool trajectory or LLM-as-judge scoring, or build custom checks in the visual designer. Pick the evalset to score against and set the target pass rate.
- Run the evaluation. The report shows pass or fail per case, and the Evaluation History view tracks pass rate across runs so you can correlate regressions with the changes that caused them.
When to use evaluations
| Use evaluations when... | Look elsewhere when... |
|---|---|
| You're about to change instructions, tools, or the model and want a regression check. | You need a single deterministic unit test for a pure function. |
| You want a baseline of agent quality you can track across runs and commits. | You're debugging one specific failed run. Use Observability. |
| You need to verify safety and refusal behaviour before shipping. | You're still prototyping and haven't picked the agent's tools yet. |
What's next
- Create evalsets — Capture real chats and curate them into a golden dataset.
- Create evaluations from templates — Configure a prebuilt evaluation template in a form.
- Create custom evaluations — Build your own checks in the visual designer.
- Run evaluations — Execute runs and review the report and history.