Reliable agent executions

Traditionally, executing an action requires writing code. However, prompting a Large Language Model (LLM) has emerged as a strong competitor. For many use cases, asking an LLM to carry out an action requires no prior preparation and feels entirely natural. A great example of this kind of problem is the tau2 benchmark.

There is a lot of work on LLM-based execution and we have seen 80-90% successful completions on the tau2 benchmark (e.g., https://artificialanalysis.ai/evaluations/tau2-bench).

Can we depend on LLM based executions? What if an LLM hallucinates or misses an instruction? While these concerns are valid, significant progress is being made. Among those approaches are improved models, fine-tuned models, and improved techniques for managing memory and context. While those approaches have made significant progress, they will not eliminate the risk of hallucinations.

In our recent paper at CAIS, we take a very different approach to solve this problem by building a safety net for the agent that provides much stronger guarantees. We proposed an approach where we let the LLM execute things as usual while keeping a recovery log, which we used to provide guaranteed recovery in case of a failure.

For any execution that includes reversible or compensable operations with detectable failures, this can guarantee that no bad things can happen (no unexpected side effects are left). Here, reversible operations reset the world to a new state equivalent to the one obtained when the operation is never performed, while compensable means the operation can’t be reversed but can be compensated. For example, providing future credits for late delivery.

How does it work?

Let's consider travel booking as an example. If a travel booking process fails in the middle, we do not want it to leave some hotels or airlines booked.

When we use agents to carry out actions (booking travel), those actions can fail, leading to unexpected side effects that agents need to compensate for and loop through until goals are met and no side effects remain. However, due to hallucinations or context rot, the agent can lose track of some actions and leave unexpected side effects.

Figure 1: How does RAC recovery work?

As shown in Figure 1, RAC introduces predictability by allowing the agent to execute as usual while maintaining a recovery log of all actions. If execution is successful, we do nothing, but if it fails, we take over. We first try to recover by retrying if the failure is transient. If not, we try to find an alternative. If both fail, we start rollback.

Rollback uses the recovery log to identify compensating actions and then applies them to each successfully executed action. We support finding compensations from configurations, from MCP, or an LLM.

Then, we execute the compensation action for each action in the reverse order in which they happened. What happens after compensation depends on the agent graph logic. For example, the agent graph may retry, try an alternative, or report to the user based on its logic.

RAC keeps a recovery log using an extension provided by the agent framework. So you can enable RAC for most agent frameworks (e.g., LangChain, CrewAI) with very few or no changes to your existing agent code.

To evaluate RAC, we compared it against SagaLLM, the planning-based state-of-the-art (SOTA), ReAct, and ReAct with prompts (prompt asking not to leave any side effects) using the following three benchmarks.

1. Tau2 benchmark - reasoning and tool use with disruptions.
2. REALM-Bench - complex, dynamic real-world planning scenarios.
3. Three new problems that include errors not mentioned in the prompt and impossible scenarios.

RAC outperformed other options across all three benchmarks. With Tau2, ReAct did 40-90% while SagaLLM and RAC did 97+%, but used more tokens. REALM-Bench - all three are mostly successful, but SagaLLM used 3-20X more tokens due to replanning loops. With three New problems, ReAct did 3/9, SagaLLM did 4/9, and RAC did 8/9 runs correctly.

Key learnings from the paper

While sayings like “failing to plan is planning to fail” suggest planning is better than ad hoc greedy executions (e.g., ReAct), the reality is more nuanced. If correct, planning can be faster, but when planning fails, replanning loops can be very expensive. When planning has to handle all errors, it becomes complex and harder to get it right.

For example, not all possible errors may be mentioned in the prompt, and anticipating them is hard, as we saw when examining benchmark results in depth. We believe there is a pre-planning vs. ReAct tradeoff, which we plan to explore as future work. For example, if we are planning a trip to Europe, muscle-level planning (planning what each muscle should do in the trip) does not work. We need to do high-level planning first, and muscle-level planning can be done on demand.

Second, why did RAC (which uses ReAct underneath) outperform ReAct? We believe this is because a) the recovery log provides a clear view of the progress so far, which helps battle context rot, and b) with RAC, ReAct does not need to worry about failures, which simplifies the problem.

Third, current benchmarks like Tau2 do not capture the nuances such as planning vs. ReAct trade-offs. The three problems we introduced capture some of those scenarios. Extending the benchmarks is another direction we plan to explore.

What does this mean?

Instead of trying to control what the agent does, ReAct tries to provide a safety net that decouples failures and agent execution side effects from LLM reasoning, where RAC tracks all activities via a transaction log and ensures that all side effects are compensated.

This provides the predictability and stability we expect from a robust system. Furthermore, it frees the reasoning loop from worrying about errors, simplifying its reasoning and increasing its effectiveness.

Therefore:

1. If you want to run reliable agent-based executions, RAC ensures no unexpected side effects will be left with any execution that includes reversible or compensatable operations with detectable failures.
2. You can integrate RAC with most agent frameworks with minimal additional work.
3. You can use RAC with minimal code changes to your current agents.

We believe the best RAC configuration is to use MCP for finding compensating operations with LLM-based data mapping.

Where it might not work

If actions are reversible and errors are detectable, we can use RAC.

What do we do when an action is not reversible? If the impact is small, we can handle it similarly to compensation (e.g., give the customer some credits) and use RAC. If actions are not reversible and the impact of compensating the action is deemed to be significant, we should not unleash agents on such problems, or at least we need human approval.

What happens if an agent gives a wrong answer, but there is no error? (e.g., non-optimal itinerary). Then we do not know. RAC does not solve this problem, but RAC can work with any techniques for detecting such semantic failures and quality control.

Remember that even reversible actions may incur costs. E.g., canceling a flight usually incurs some fees. When such costs are significant, we need to consider this in overall planning. RAC does not handle this case at this time.

You can find more details in the paper and the slide deck. Code can be found at https://github.com/wso2incubator/research-rac. The CAIS talk is on the ACM YouTube channel: https://www.youtube.com/watch?v=ZgV8CezSNcs&list=PLTih9KsCymcg&index=23

If you are working on similar problems, we would love to hear from you.

References

1. Perera, Srinath, et al. "Robust Agent Compensation (RAC): Teaching AI Agents to Compensate." Proceedings of the ACM Conference on AI and Agentic Systems. 2026.
2. Chang, Edward Y., and Longling Geng. "SagaLLM: context management, validation, and transaction guarantees for multi-agent LLM planning." arXiv preprint arXiv:2503.11951 (2025).
3. Barres, Victor, et al. "$\tau^ 2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment." arXiv preprint arXiv:2506.07982 (2025).