Managing AI Agents at Machine Speed: Closing the Control Gap

In July 2026, OpenAI disclosed that AI agents running cybersecurity evaluations had gradually moved beyond their assigned tasks and eventually compromised parts of Hugging Face's production infrastructure. Over several weeks, the agents established unintended communication channels, gained internet access through infrastructure that was meant to isolate them, and reached external systems. Earlier warning signs had appeared, but the broader pattern was recognized only after a security alert triggered a larger investigation [1].

This isn’t the first reported instance of agents moving beyond their assigned boundaries. 

Earlier this year, in April, an AI coding agent working in PocketOS's staging environment encountered a credential mismatch. According to the company's founder, the agent searched for another way to authenticate, found a broadly privileged API token in an unrelated file, and used it to delete the company's production database volume and its volume-level backups. The destructive API call took about nine seconds [2].

The first failure developed gradually as agent behavior moved outside its intended operating boundary. The second crossed that boundary almost immediately. But they reveal the same governance weakness: organizations cannot rely on people to watch agents continuously, and human intervention may come only after the damage is done.

The real risk is therefore not simply that an AI agent can act autonomously. It is that the agent can exercise real authority faster than the organization can detect harmful behavior, constrain that authority, or recover from the consequences.

Why agents create a governability gap

This imbalance can be described as the governability gap: the mismatch between expanding agent capability and limited organizational control [3]. Here, the term has a practical enterprise meaning:

The governability gap is the distance between the authority an agent can exercise and an organization's ability to observe, constrain, interrupt, and recover from its actions.

The gap can open through accidental failure or through behavior that moves outside the user's intent. METR is a research nonprofit that studies whether and when advanced AI systems could pose catastrophic risks. It maintains a catalog of documented incidents in which agents acted against their users' intentions [4]. These cases show that reliable model behavior alone is not enough. Once a model can decide to use tools and act autonomously, governability must also keep that autonomy within defined and acceptable boundaries.

The gap widens when an agent design allows it to act before harmful behavior is detected, reach systems outside its assigned task, or make changes that are difficult to reverse. It can widen further when no one is clearly responsible for deciding what the agent should be allowed to do or when its scope expands.

Much of enterprise governance still operates at human speed. Organizations rely on dashboards, alerts, approvals, incident calls, and management decisions. An agent can make a decision, call a tool, change a system, and continue to its next step before a person receives the first alert. Gartner similarly warns that autonomous agents can act at a scale and speed that outpace human oversight [5].

Closing the governability gap requires six connected capabilities:

1. Agent identity and task-bound authority  

2. Limits on agent actions  

3. Observability and behavioral monitoring across the run  

4. Safe containment and recovery  

5. Clear ownership and accountability

6. Continuous learning from testing and operational evidence to strengthen the controls

An agent control plane coordinates these capabilities across agents, frameworks, runtimes, and environments.

Define agent identity and task-bound authority

The organization must explicitly define the boundaries within which an agent may operate. Agent authority has two layers:

• Agent identity and access controls establish which agent is acting, the credential it uses, and the maximum resources and operations it can access.
• Task-bound authority narrows that access to what is legitimate for the current assignment or run.

Possessing a permission does not make every use of it legitimate. The current task determines which permissions the agent may use and for what purpose. In the PocketOS incident, the agent was working on a staging task when it encountered a credential mismatch. Deleting production storage was outside that task [2]. The token made the deletion technically possible, but the task did not make it legitimate.

OAuth security guidance recommends binding tokens to their intended sender and limiting privileges to what a particular use case requires [7]. For agents, that maximum access is only the starting point. The current task must further narrow what the agent may do. Task and run limits can cover resources, operations, environments, parameters, duration, cost, retries, and tool-call rates.

These limits define what the agent may do for the task. Enforcement applies them to each action the agent proposes.

Enforce limits on agent actions

Every request from an agent to a tool or external system should be checked against the agent's authority before it proceeds. The check must answer a specific question: Is this agent allowed to make this request, for this task, in this environment, now? 

The answer should not depend on the proposed action alone. Repeated failures, earlier policy denials, or changes already made to external systems may affect whether the next action should be allowed [8].

Prompt instructions and internal safeguards can guide agent behavior, but organizations should not rely on the agent alone to enforce its own limits. When it encounters an obstacle, it may choose an unexpected route in an effort to complete the task. Authorization must therefore be applied independently to the request the agent actually sends.

Tool calls are one of the primary ways an agent acts on external systems. A gateway is therefore a natural enforcement boundary for these requests. If an action cannot pass through a gateway, the same check must occur where it is invoked. Every route to an external system must be covered. Credentials for those systems should be injected or supplied at the enforcement boundary rather than exposed to the agent runtime.

The enforcement decision should consider:

• the agent’s identity and credential
• the task-bound authority granted for the current run
• the proposed operation, its parameters, and its target environment
• relevant earlier actions and their outcomes
• the expected impact, whether the action can be reversed, and whether approval is required.

Checks against clearly defined limits, such as permitted operations, resources, environments, and parameter ranges, should run automatically. A request outside the task's authority should be blocked. When human judgment is required, the request should be paused for approval while the run's state is preserved.

In the reported PocketOS incident, the agent encountered a credential mismatch, searched an unrelated file, found a broadly privileged token, and used it to delete production storage through an infrastructure API [2]. The reported incident shows no evidence of an independent enforcement layer that checked the proposed action against the assigned task before execution, allowing the agent to use the token it found to act directly on production.

Action-level enforcement decides whether the next request may proceed. Behavioral monitoring asks whether the run as a whole is still operating as intended. A sequence of permitted actions can still form a failing pattern.

Figure 1: Action-level enforcement governs each proposed action, while behavioral monitoring identifies patterns across the run.

Monitor behavior across the run

Organizations often discover a failing run only when its consequences become visible: an unexpected bill, corrupted records, an outage, a customer complaint, or missing data. By then, harm has already begun.

Some failures are operational: an agent may misunderstand its task, become stuck, or repeat actions without making progress. Others arise when it moves beyond its assigned task or objective. Behavioral monitoring uses traces and metrics from the agent’s LLM and tool interactions to identify these patterns across the run.

Agents can go off course in many ways. The following five patterns provide a practical starting point because they cover five observable dimensions: scope, objective, progress, response to controls, and scale.

• Scope drift: The agent operates outside the boundaries of its assigned task or delegated authority.
• Objective divergence: The agent begins pursuing an outcome different from the objective it was given.
• No-progress loop: The agent repeats actions without producing a meaningful state change.
• Guardrail probing: After an action is blocked or a path fails, the agent tries alternatives that approach the same control boundary.
• Uncontrolled fan-out: The agent launches sub-agents or invokes tools at a rate or scale outside the task's expected operating pattern.

No single signal proves that an agent has gone off course. Many tool calls, an unusual sequence, or a temporary detour may still be consistent with legitimate execution. Such behavior must be evaluated against the task, task-bound authority, execution history, and expected operating pattern. An action-level policy may respond to an individual action, while behavioral monitoring shows whether signals repeat, intensify, or combine across the run.

The OpenAI/Hugging Face incident combined several patterns. During cybersecurity evaluations, agents created unauthorized communication channels and gained internet access. After OpenAI’s initial mitigations, they found new ways to restore both and later accessed unrelated third-party systems [1]. Some agents also shifted from their assigned tasks to seek capabilities for the agent collective. The incident illustrates scope drift, objective divergence, and repeated attempts to cross control boundaries.

A no-progress loop presents a different failure mode. One anonymized vendor report describes an agent retrying the same failed operation hundreds of thousands of times, generating $47,218 in model usage [9]. Uncontrolled fan-out creates another risk: rapid delegation or parallel tool use can expand execution faster than it can be observed or controlled [10].

Detecting these patterns creates an opportunity to intervene before the consequences escalate. The next challenge is stopping the run safely and dealing with what has already happened.

Contain harm and recover safely

Stopping or slowing an agent can prevent further action. It does not undo changes the agent has already made.

An agent may be stopped between updates to several systems or while an external operation is still in progress. This can leave partially completed work, duplicate execution, inconsistent records, or a failure that recurs when the agent resumes.

Intervention must therefore separate containment from recovery. Detection and immediate containment can operate automatically at machine speed, while recovery requires human judgment.

Figure 2: Detection and containment limit further harm automatically; stabilization verifies actual system state before human-authorized recovery.

Containment should match the immediacy of the risk:

• Throttle: Slow the agent when its behavior requires closer observation but does not present an immediate danger.
• Suspend: Pause execution, preserve the current state, and escalate the run for human review.
• Quarantine: Isolate the agent and revoke its authority when misuse or compromise is suspected, while preserving evidence.
• Terminate: End the run when continued execution presents an immediate danger.

A kill switch belongs here. It is an emergency containment mechanism, not a detection system, authorization policy, or recovery strategy.

Once the risk has been contained, recovery can begin:

• Compensate: Reverse completed actions or reconcile systems that no longer agree.
• Resume from a checkpoint: Continue from a known safe state.
• Retire the run: End the run when its state can no longer be trusted.

To stabilize the situation before recovery, the team must determine which actions were attempted, authorized, completed, or left in progress, reconcile any partial changes, and verify the affected system’s actual state. This is why the gateway or equivalent external enforcement boundary should produce a tamper-resistant audit trail of policy decisions, approvals, attempted actions, and observable outcomes. Recovery may also require verifying the affected system’s final state.

Safe recovery also depends on durable checkpoints, operations that can be retried without duplicating work, fast credential revocation, and ways to reverse changes. These capabilities must exist before an agent is trusted with consequential work. They also need someone accountable for keeping them ready and deciding how they should be used.

Give every production agent a named, accountable owner

Autonomy may remove the need for a person to approve every action, but it cannot remove organizational accountability for the authority granted to an agent. The National Institute of Standards and Technology (NIST) AI Risk Management Framework calls for clear roles, responsibilities, and human-AI oversight [6]. Every production agent should therefore have a named owner accountable within the organization for its purpose, delegated authority, operation, and outcomes.

In the operating model proposed here, the owner and builder are separate roles assigned to different people or teams. The owner defines the agent’s purpose, expected outcomes, and operating boundaries within organizational policy. The builder implements and maintains its logic and integrations within those boundaries. The organization’s governance function defines common policies, risk classifications, and minimum control requirements. Through the agent control plane, it maintains a system of record for agents, coordinates incident response, and verifies that agents remain within governed boundaries.

These responsibilities continue throughout the agent’s lifecycle. Before deployment, the owner ensures that approval requirements, signs of failure, and containment and recovery plans are defined and tested. After deployment, the owner and governance function use evidence from evaluations and production runs to determine whether the agent remains within its intended boundaries and whether its controls still work. The builder implements any required changes as the agent, its tools, and operating conditions evolve.

Continuously learn and strengthen the controls

The controls around an agent cannot remain static. Observability signals, enforcement decisions, user feedback, near misses, incidents, and recovery attempts reveal how agents and safeguards behave in practice. Learning from this evidence goes beyond measuring agent accuracy. Its purpose is to find gaps in authority, enforcement, monitoring, intervention, and recovery.

Some gaps can be found through controlled testing. Others appear only with real users, data, tools, and operating conditions. Both the agent and its environment will also change. Teams must therefore test and strengthen the controls throughout the agent’s lifecycle.

Teams identify gaps in two ways:

1. Controlled testing outside production. Sandboxes, staging environments, simulations, red-team exercises, and replayed traces allow teams to challenge the agent and its controls without exposing live systems. This testing should happen before launch and whenever the agent or its operating environment changes.

2. Learning from production evidence. Behavioral monitoring detects patterns during an active run. Production evaluations analyze traces from live runs, together with user feedback, policy denials, blocked actions, near misses, incidents, and recovery outcomes. Because this analysis happens outside the execution path, teams can investigate uncertain patterns without delaying or blocking the agent.

Reviewing completed runs in either setting can expose control failures. Prompted by OpenAI’s Hugging Face incident, Anthropic searched 141,006 of its own cybersecurity evaluation runs for similar failures. It found three incidents in which Claude models gained unauthorized access to real systems through internet paths left open by misconfiguration [11]. The review exposed gaps in environment containment and real-time monitoring, not simply incorrect model outputs.

Whether discovered through controlled testing or production evidence, a finding should not become a new guardrail immediately. Teams should first reproduce the gap and test whether the proposed safeguard addresses it without blocking legitimate activity. Once validated, the finding can strengthen authority limits, enforcement policies, monitoring, approval requirements, intervention, or recovery. The scenario should also become part of future testing.

NIST similarly recommends testing AI systems before deployment and regularly while they are operating [6]. For agents, this testing must cover the controls around the agent, not only whether the agent produces correct outputs.

This creates a continuous feedback loop: test the controls safely, learn from production evidence, validate improvements, and repeat.

The test for sufficient controls

Before giving an agent consequential authority, an organization should be able to:

• identify the agent and define its authority for the current task
• block or escalate unauthorized actions before they execute
• detect when a sequence of actions is going off course
• contain a failing run and recover affected systems safely
• assign a named owner accountable for the agent’s operation and outcomes 
• use evidence from testing and production to strengthen its controls.

Together, these capabilities must be implemented as an operating architecture, not left as policies or principles. An agent control plane coordinates identity, authority, lifecycle, policy, and evidence across the organization. Governed ingress and egress gateways then enforce those controls where interactions enter and leave managed agents.

Figure 3: An agent control-plane architecture that coordinates identity, policy, lifecycle, and evidence while governing every managed path into and out of AI agents.

The practical test is whether the organization can observe how an agent uses its authority, constrain its actions, intervene when necessary, and recover from the outcome. If it cannot, the agent’s authority should be reduced until the missing controls are in place. WSO2 Agent Manager is one example of a complete agent control plane offering these capabilities. 

Passing this practical test is the condition for scaling agentic systems with confidence. For the broader architectural rationale behind this approach, read Trusted AI Governance: An architectural thesis for governing agentic systems.

---

References

 

1. OpenAI, “The Hugging Face incident and the road ahead,” August 26, 2026.
2. Tom Carter, “A founder says Cursor's AI agent deleted his startup's database, causing chaos for customers,” Business Insider, April 28, 2026; and Rich Stanton, “Here we go again: AI deletes entire company database and all backups in 9 seconds, then cheerfully admits ‘I violated every principle I was given’,” PC Gamer, April 28, 2026.
3. Mitch Ashley, “Futurum Agent Control Plane Framework: A Reference Model for Production AI Agents,” The Futurum Group, April 3, 2026.
4. METR, “Documented AI Agent Incidents,” last updated May 19, 2026.
5. Gartner, “Gartner Says Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure,” May 26, 2026.
6. National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework (AI RMF 1.0),” NIST AI 100-1, January 2023; see also the NIST AI RMF Core.
7. Torsten Lodderstedt, Joseph Bradley, Andrey Labunets, and Daniel Fett, “Best Current Practice for OAuth 2.0 Security,” RFC 9700, BCP 240, January 2025; see Sections 2.2–2.3.
8. Maurits Kaptein, Vassilis-Javed Khan, and Andriy Podstavnychy, “Runtime Governance for AI Agents: Policies on Paths,” arXiv:2603.16586, March 17, 2026.
9. Tijo Gaucher, “Stop the $47K Runaway Loop: Rate Limiting & Cost Guardrails for AI Agents,” Rapid Claw, April 18, 2026. This is an anonymized vendor case study; the organization and incident are not independently identified in the source.
10. Google Cloud, “Choose a design pattern for your agentic AI system,” Cloud Architecture Center, last reviewed May 28, 2026; and Microsoft, “Unmanaged Agent Sprawl & Collusion,” Microsoft Learn, updated August 1, 2026.
11. Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations,” July 30, 2026.