Trusted AI Governance for the Agentic Enterprise

AI governance is often discussed as a problem of model safety, responsible AI, or output quality. Those are important concerns, but they are no longer enough as enterprises move from AI that generates content to agents that can take action. An agent can access enterprise data, call APIs, use tools, trigger workflows, communicate with other agents, and make decisions based on changing context. At that point, the governance problem changes.

We are no longer only concerned with whether the model produced a good answer. We also need to establish whether the agent acted within valid authority, used data and tools appropriately, followed policy, and produced enough evidence to reconstruct what happened.

That makes trusted AI governance an architecture problem.

The governance model has to change

Most enterprise software is built around deterministic behavior. We define a contract, implement it, test it, deploy it, and monitor whether the system continues to behave according to that contract. An API has a defined interface, a workflow follows a known sequence, and an integration performs a defined transformation. There can be failures and bugs, but the expected behavior can generally be specified before the system runs.

Agentic systems are different. An agent reasons based on its instructions, context, memory, available tools, model behavior, and the results of previous actions. Change any of those, and the path it takes may change. Two executions of what appears to be the same task may not follow exactly the same path. This creates an inversion in governance.

With deterministic systems, governance can start with the specification and verify that the implementation follows it. With probabilistic agentic systems, we cannot completely specify every possible output or execution path in advance, so governance has to move closer to the point where intent becomes action.

The focus shifts from verifying the correctness of the result to verifying whether the action was permitted, whether the authority was valid, whether the data and tools were used appropriately, and whether the action stayed within the intended context and policy boundaries.

Pre-deployment evaluation still matters. Models, prompts, tools, policies, and workflows should be tested before release. But that cannot carry the full governance burden because the runtime path itself becomes part of the governed surface.

Trust is an architectural property

Trusted AI should not depend on confidence in the model alone. Agentic systems are probabilistic, so trust has to come from the architecture that governs how agents are identified, authorized, constrained, observed, and held accountable.

In practice, that means agent behavior needs to be inspectable, attributable, and bounded. Inspectable means the enterprise can reconstruct what happened, including the context used, tools called, data accessed, policies applied, and where human intervention occurred. Attributable means actions can be tied to a specific agent and a clear chain of delegated authority. Bounded means the agent operates within defined limits across data, tools, actions, cost, jurisdiction, and human approval.

This makes discovery, identity, delegation, authorization, policy enforcement, and evidence foundational to trusted AI governance. Agents need to become first-class actors in the enterprise architecture, with explicit identities and traceable authority. From there, governance can be enforced both within the agent implementation and through the infrastructure around it.

Two enforcement models are needed

I see two complementary ways to enforce governance. The first is inline governance, where controls are implemented directly in the agent or application logic. This is useful because the application understands local business context, workflow state, domain rules, and the intent behind the action. The second is infrastructure-enforced governance, where controls sit outside the individual agent implementation at shared boundaries, such as gateways, runtimes, identity infrastructure, proxies, service meshes, and agent-to-agent communication layers.

Both approaches are necessary. Inline governance without infrastructure enforcement becomes fragmented because every team has to independently implement the right controls, keep them updated, and interpret enterprise policy consistently. Infrastructure enforcement without inline context can be too coarse because shared infrastructure may know what resource is being called but not always why the application is calling it.

Therefore, the two models have to work together, and infrastructure enforcement also needs to be parameter-aware. Knowing that an agent is allowed to invoke a tool is not enough. Governance needs to understand what the agent is trying to do with that tool. An agent may be allowed to call a payment API but not to issue a refund above a certain amount. It may be allowed to send email but not to send regulated data to an external recipient. It may be allowed to access customer records but only the fields required for the task. The risk is usually not in the existence of the tool call. It is in the parameters of the action. This is where broad permission becomes bounded action.

Compliance has to become executable

Compliance is often treated as something that sits around the system. Policies are written, controls are documented, reviews are performed, and audits verify whether the organization followed them. That model becomes less effective when autonomous systems are making decisions and taking actions continuously.

Regulatory obligations, internal policies, data residency rules, risk thresholds, human approval points, and segregation-of-duty requirements increasingly need to become part of the runtime control system. In other words, policy cannot stop at documentation. It has to be translated into enforceable controls at the points where agents access data, invoke tools, delegate authority, communicate with other agents, or take business actions.

This is also why policy authorship and enforcement need to be separated cleanly. A central governance function may define the policy, but enforcement has to happen close to where the action occurs. That turns compliance from a periodic review activity into an architectural property of the system. It also means evidence becomes part of governance. If we cannot completely determine the execution path before an agent runs, we need enough information after it runs to understand what happened. Observability therefore becomes more than an operations concern.

The system should be able to show what the agent did, what authority it used, what tools were invoked, what data was accessed, what policies applied, what decisions were made, and where human judgment entered the process. This matters for security and compliance, but also operationally. When something goes wrong, there needs to be a reconstructable chain of action, authority, policy, and evidence. That is also what makes compliance provable. It is not enough to say that a policy exists. The enterprise needs evidence that the policy was applied to the action that actually occurred.

Human oversight should be designed in

Human oversight is often treated as a fallback when automation fails. For agentic systems, I think it needs to be designed as an explicit control point. Not every action requires human approval, nor should it. The goal is not to keep a human in every loop. The goal is to identify the points where risk, business impact, regulation, cost, or irreversibility require human judgment.

A low-risk action may proceed autonomously, a higher-risk action may require approval, and some actions may never be appropriate to delegate fully. When approval is required, the human should also have enough context to make a meaningful decision, including the proposed action, the authority being used, the relevant policy, and the expected consequence. In other words, human oversight should be part of the architecture, not an exception bolted onto it later.

Governance has to be architectural

None of this removes the need for model evaluation, responsible AI practices, red teaming, safety testing, or organizational governance. They all remain important, but they sit inside a larger system. As agents move from answering questions to taking actions across enterprise systems, trusted AI governance has to address discovery, identity, delegation, inline controls, infrastructure-enforced and parameter-aware controls, runtime safety, compliance policy, human oversight, evidence, and economic limits as one connected architecture.

The goal is not to make probabilistic systems deterministic. We cannot. The goal is to build an architecture where probabilistic systems can operate within the deterministic boundaries of authority, policy, risk, and accountability. That, to me, is the foundation of trusted AI governance.

Read the full architectural thesis: Trusted AI Governance: An architectural thesis for governing agentic systems. https://github.com/wso2/reference-architecture/blob/master/trusted-ai-governance.md