Agent Platform Architecture: Orchestration to Observability

The first agent a team ships usually works. It runs on one framework, calls one model, and uses a handful of tools that its developer wired up by hand. Then a second team builds an agent on a different framework. A third team wants two agents to hand work to each other. Within a quarter, nobody can say which agents are running, which models they call, or what data they touched yesterday.

That is the point where agent platform architecture becomes its own discipline. In 2026 the building blocks are maturing quickly: MCP for tool access, A2A for agent-to-agent calls, and OpenTelemetry for tracing. The architecture that ties them together is still settling. This guide maps that architecture using three planes (build, orchestrate, and control). It then works through the layers each plane depends on: orchestration patterns, task routing, memory, observability, and identity.

Why Most Agent Platforms Never Leave Proof of Concept​

Agents combine non-deterministic behavior with broad access to real systems, and that combination is hard to carry from pilot to production. Pre-agentic platforms were not designed to handle both at once. A demo can tolerate that combination. Production cannot, because it needs identity, policy, tracing, and evaluation around every agent.

In a June 2025 forecast, Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027. The reasons it gives are escalating costs, unclear business value, and inadequate risk controls. The same release notes that most projects at the time were early experiments or proofs of concept. This is a forecast, not a measured failure rate, and it does not name architecture as the cause. This guide treats those three risks as design inputs. Cost visibility, evaluation, and runtime risk controls are platform concerns, and the sections below cover each one.

Part of the response is architectural. Existing distributed applications already need runtime controls such as authorization, monitoring, concurrency control, and incident response. Agents inherit all of that and add variable action selection, plus model and tool behavior that can change between runs. An agent decides its own next step at runtime. It may call a tool nobody expected, or pass a customer record to a model in another region. When that agent also holds credentials to production systems, the platform has to answer questions that pre-release testing alone cannot answer. Who is this agent? What is it allowed to do? What did it actually do, and was the result any good?

Teams that skip those questions rarely fail at the model layer. They fail when a security review asks for an audit trail, or when finance asks which team spent the inference budget. The APIs underneath matter as well, because they are the API layer these agents ultimately call. Governing agents while exposing poorly scoped APIs only moves the risk.

The Three Planes of an Agent Platform: Build, Orchestrate, Control​

Forrester's three-plane model splits the enterprise agentic stack into build, orchestrate, and control. It is an analyst reference model, not a formal interoperability standard. It is a useful map because each plane answers a different question, and each tends to be owned by a different team.

Build plane​

This is where agents are created. It covers model access, agent frameworks such as LangGraph or CrewAI, tool integrations, prompts, and evaluation pipelines. Developers own it, and most enterprises will run more than one framework here.

Orchestrate plane​

This is where agents are embedded into business processes and workflows, alongside non-agentic services. Multi-agent coordination lives here. So do long running workflows, event driven triggers, and the points where a human review step is required.

Control plane​

This is where agents are inventoried, governed, and observed. Forrester's argument is that governance has to sit outside the other two planes. Only then can it provide independent visibility and consistent policy when runtimes behave unpredictably.

That last point has a direct consequence for traffic design. The control plane is not a hop that every request passes through. It sets identity, policy, and budgets, then receives telemetry back. Enforcement happens in the data plane, usually at a gateway that sits on each traffic path. Being outside the request path does not remove runtime dependencies. Gateways still rely on the control plane for credentials, token validation, and current policy. Its availability and policy freshness therefore still affect live traffic.

The control plane pushes policy and credentials down and pulls telemetry back up. External agents are governed only on calls routed through AI Gateway, and traced once instrumented.

The diagram above separates three flows. Inbound calls arrive from users, applications, or other agents. Outbound LLM calls and outbound tool calls leave the agent on separate routes, each with its own policy. Separate routes are not functionally independent, though. An agent often needs a model response before choosing a tool, so a slow or failed LLM call can stall the tool path too. The control plane pushes policy and credentials down to the gateway and agents, and pulls traces and metrics back up.

Orchestration Patterns: How Multiple Agents Actually Coordinate​

The first architectural decision is whether you need a multi-agent system at all. A single agent with well-designed tools is easier to test, trace, and secure. Teams usually move to specialized agents for one of three reasons. The task spans domains with different data access rules. Prompts and tool lists grow too large for one agent to handle reliably. Or different steps need different models.

Once you split work across agents, the coordination pattern determines how failures spread and how hard the system is to debug.

Pattern	How It Works	Best For	Key Risk
Sequential	Each agent processes the output of the previous one in a fixed order	Pipelines with clear stages, such as extract, enrich, then summarize	One bad output propagates through every later stage
Concurrent	Several agents work on the same input in parallel, and results are merged	Independent analyses that a final step combines	Conflict resolution between contradictory results
Handoff	An agent transfers control of the conversation to a more suitable agent	Triage flows, such as routing a support request to billing	Lost context at each transfer
Group chat	Agents share one conversation thread and take turns contributing	Review and debate tasks with a human moderator	Unbounded turns and rising token cost
Hierarchical (supervisor)	A supervisor agent plans, delegates subtasks to workers, and assembles the result	Complex tasks with decomposable subtasks	The supervisor becomes a single point of failure and a bottleneck
Peer-to-peer	Agents call each other directly without a central coordinator	Cross-team or cross-organization agents that each own a domain	Hard to trace, and loops are easy to create

Three of the six coordination patterns, with the risk each one carries.

Most production systems combine patterns. A supervisor might run a sequential pipeline for one subtask and fan out concurrently for another. For a worked example of the supervisor pattern, see the multi-agent calendar assistant case study. Peer-to-peer coordination is becoming more practical as agents publish their capabilities through A2A Agent Cards. That makes discovery easier. It also widens the trust boundary, because an agent can now receive work from agents it has never seen before. Pick the simplest pattern that fits the task, and add coordination only when a measured problem requires it.

Task Routing and the Control Plane​

Routing in an agent platform happens at two levels, and conflating them causes trouble.

The first level is agent selection. Task intake decides which agent, or which workflow, handles a request. In a hierarchical system the supervisor makes that decision. In a handoff system the current agent does. This logic belongs to the orchestrate plane, because it encodes business rules about who does what.

The second level is model and tool selection. Once an agent is running, it calls models and tools. Which model it may call, at what cost, and with what data controls is a policy decision. That decision should not live inside each agent's code. If it does, every framework team reimplements budgets, fallbacks, and data rules differently. A change in provider then means a change in every agent.

The cleaner design keeps routing policy in the control plane and enforces it at a gateway on the outbound LLM path. The agent asks for a capability. The gateway picks the model based on cost, latency, capability, or data residency. LLM routing covers those strategies in depth.

Centralizing this is not always worth it on day one. The useful triggers are qualitative:

• Multi-model usage: More than one provider or model family is in play, and you want failover or capability-based routing without code changes.
• Spend attribution: Finance needs inference cost broken down by team, agent, or product.
• Regulated data in prompts: Prompts or tool results contain personal, financial, or health data that needs masking or regional routing.
• Production scale: Several teams ship agents on different frameworks, and policy drift between them has become a real risk.

Conflict resolution also belongs in this discussion. When two agents produce contradictory outputs, decide in advance whether a supervisor, a deterministic rule, or a human makes the call. Leaving it to whichever agent answers last is not a policy.

State, Memory, and Context Across Agent Sessions​

Memory architecture is where agent systems most often drift from correct behavior without anyone noticing. It helps to separate three kinds of state.

• Short term memory is the working context of a single session: conversation history, intermediate tool results, and the current plan. It is bounded by the model's context window, so long sessions need summarization or truncation. Each of those choices drops information.
• Long term memory persists across sessions, often as user preferences or documents in a vector store. It makes agents more useful over time, and it turns every write into a potential security event.
• Shared state is what multiple agents read and write together, such as a task board or a common case record. State management here needs the same discipline as any distributed system: clear ownership, versioning, and concurrency control.

For the memory types and storage choices in more depth, see AI agent memory.

The security side of memory is now well documented. The OWASP Top 10 for Agentic Applications lists Memory and Context Poisoning as a distinct risk. An attacker plants misleading content in memory or a retrieval store, and the agent acts on it in a later session. Context pollution is the non-malicious version of the same problem. Stale or irrelevant context gradually degrades agent behavior.

The controls are easy to state and hard to retrofit:

• Scope memory by user, tenant, and task, so one session cannot read another's context.
• Record the source of every memory write, so suspicious entries can be traced and rolled back.
• Set expiry on long term memory instead of keeping everything indefinitely.
• Treat retrieved memory as untrusted input, and apply the same guardrails you apply to user prompts.

Observability and Evaluation: Seeing What Agents Actually Do​

Observability is a required platform layer, not a debugging convenience. An agent's behavior is the sequence of decisions it made, so request logs alone are not enough. You need the full trace from the inbound request, through each model call, to each tool invocation.

OpenTelemetry is the common foundation. Its GenAI semantic conventions define agent operations such as invoke_agent and execute_tool. Traces from different frameworks can then be read in one place. The conventions are still maturing, so expect some framework-specific attributes for now.

Traces show what an agent did. Evaluation shows whether it did it well. A production setup typically runs evaluations in two places. Offline evaluations run in CI/CD against curated test sets before a new prompt or model version ships. Online evaluations score sampled production traffic against criteria such as task completion, groundedness, or policy compliance. Both feed the same question: is this agent version good enough to keep running?

Guardrails sit next to this layer. They act on individual requests in real time, while evaluations judge behavior across many requests. Root cause analysis needs both. A guardrail block tells you a request was stopped. The trace and evaluation history tell you why the agent produced it in the first place. For the agent side of this telemetry, see agent observability. For the gateway side, see AI gateway observability.

Security, Identity, and Governance for Agent Platforms​

The central security question for an agent platform is blast radius. If an agent is manipulated through prompt injection or a poisoned tool description, how much damage can it do? Identity and permission design narrow it most directly. Egress controls, data permissions, business limits such as refund caps, and approvals bound to a specific action also shape it.

Start with identity. Many early agents run on a shared service account or on a user's borrowed token. Both make it hard to tell what the agent did versus what the person did. The OpenID Foundation whitepaper on identity management for agentic AI recommends treating agents as first-class citizens in identity and access management. That means each agent gets its own lifecycle, governance policies, and accountability. One of its co-authors, Ayesha Dissanayaka, is Lead Architect for Agent Identity at WSO2.

With a distinct identity in place, least-privilege agent design becomes enforceable. A narrow credential still authorizes any harmful action inside its scope, so combine it with the controls below. Permission models for AI agents covers them in more detail.

• Scope by task, not by agent. An agent that drafts refunds should not hold the permission to issue them. Put a human approval step or a separate agent with narrower rights in between. Bind each approval to the specific action and amount so it cannot be reused for another request.
• Separate inbound and outbound credentials. The credential a client uses to call the agent is not the credential the agent uses to call tools. Mixing them creates confused-deputy problems.
• Use short-lived tokens with an explicit revocation path. A short lifetime limits how long a stale or stolen token works, but it does not reject the next request on its own. If suspension must take effect on the next request, the gateway needs token introspection, a denylist, or a fresh policy check on every call. Otherwise, document the maximum remaining validity window.
• Authorize per tool. Role based access control at the tool level limits what a hijacked agent can reach.

The OWASP Top 10 for Agentic Applications is a sound baseline for threat modeling. It was developed with more than 100 industry experts and covers risks such as Agent Goal Hijack, Identity and Privilege Abuse, and Cascading Failures. Tool-level threats, such as poisoned tool descriptions, deserve a separate review of their own.

What Production Teams Get Wrong (and a Framework for Getting It Right)​

Production failures in agent systems look similar across industries and frameworks. Four come up repeatedly.

Context drift: Agents degrade over long sessions as summaries lose detail and stale memory builds up. The answers stay confident, so the drift goes unnoticed until a user complains.

Unreliable tool invocation: Agents retry failed tool calls, sometimes with slightly different arguments. Without idempotent tool design, a retry can create a duplicate order or send a second email. Error handling has to assume that any call may run twice.

Fragmented observability: Each framework ships its own tracing format, so incidents that cross two agents need two tools and manual correlation.

Rework on every new workflow: Each team rebuilds authentication, guardrails, and cost tracking for its own agent, and the policies drift apart.

These failures share a cause: platform concerns were built inside individual agents. A short checklist exposes that early. For any agent platform, current or proposed, check whether you can answer yes to each question:

1. Can you list every agent in production, its owner, and its version?
2. Does each agent have its own identity, separate from users and service accounts?
3. Are model, tool, and budget policies enforced outside the agent's code?
4. Can you trace one request across every agent, model call, and tool call it touched?
5. Do new agent versions pass defined evaluations before promotion?
6. Can you suspend a single agent immediately without redeploying others?
7. Are tools safe to retry, and are side-effecting actions gated by human review where needed?
8. Can any single agent or component fail without stopping every workflow?

A "no" on any of these is a production readiness gap, and it is cheaper to find during design than after an incident.

How WSO2 Approaches Agent Platform Architecture​

WSO2 follows the separation described above. Teams build agents with the framework that suits them, and the platform provides one control plane plus gateway enforcement on each traffic path. This is one way to implement the three-plane model. Framework-native runtimes such as LangGraph and CrewAI, and cloud services such as Amazon Bedrock AgentCore, are other options. The right fit depends on how many frameworks and clouds you need to govern.

The center of the WSO2 Agent Platform is WSO2 Agent Manager, an open source control plane for AI agents. It supports agents built with LangChain, CrewAI, AWS Strands, and Microsoft Agent Framework. It can also register agents already running in Bedrock, Azure, or LangSmith. Registration adds an external agent to the inventory, but it does not route that agent's calls through the platform. Traces appear once the agent is instrumented, and gateway policy applies only to calls the agent sends through AI Gateway. AI Gateway and Agent ID are built into it, so identity, guardrails, and observability are managed in one system.

Plane	WSO2 capability	What it covers
Build	Any agent framework, or WSO2 Integrator for low-code and pro-code agents	Agent logic, tools, prompts, and connectors
Orchestrate	Workflow logic in your agent framework (such as LangGraph), application code, or WSO2 Integrator. Agent Manager hosts the agents, and AI Gateway governs A2A calls	Workflow state and routing decisions, plus agent hosting and A2A call governance
Control	Agent Manager, with AI Gateway and Agent ID built in	Inventory, lifecycle, identity, guardrails, tracing, and evaluation

A few capabilities map directly to the sections above:

• Traffic enforcement. WSO2 AI Gateway governs LLM, MCP, and A2A traffic. On the outbound LLM path it handles multi-model routing, failover, and cost budgets that reject traffic once a budget is spent. On the outbound tool path, its MCP governance policies authenticate callers, authorize per tool, restrict the tool list, and rate limit calls. For inbound agent-to-agent calls, it publishes a governed Agent Card and applies authentication and rate limiting.
• Guardrails at several levels. Agent Manager applies guardrails at the organization, agent, MCP, and LLM levels, mapped to the OWASP Top 10.
• Agent identity. WSO2 Agent ID registers, authenticates, authorizes, and audits agents as first-class identities, with delegation policies and revocation. The identity an agent presents on outbound calls is separate from the controls that protect calls into it.
• Tracing and evaluation. Agent Manager observability is built on OpenTelemetry. It covers agents deployed through the platform and agents hosted elsewhere. Built-in and custom evaluations score agent behavior over time.
• Lifecycle control. Each agent gets a versioned path from development to staging to production. Any single agent can be suspended.

The open source Agent Manager project is licensed under Apache 2.0, per the LICENSE file in the wso2/agent-manager repository at release amp/v1.0.0 (September 2026). Its README states that agents deployed through the platform run on OpenChoreo and that instrumentation uses OpenTelemetry. You can self-host that release or use WSO2's hosted SaaS version. The Apache 2.0 license covers the open source code, not the hosted service.

Conclusion​

Agent platform architecture is a separate discipline from application platform engineering. Agents combine autonomy with broad access, and deterministic platforms were not built for that combination. The three-plane model gives that work a structure. Let teams build on the frameworks they prefer, and orchestrate agents with the simplest pattern that fits. Keep identity, policy, tracing, and evaluation in a control plane that sits outside both, with enforcement on each traffic path.

If your agents are moving from framework-level experiments toward governed production, WSO2 Agent Manager provides that control plane across frameworks and deployments. A free trial is available.

Frequently Asked Questions​

What is agent platform architecture?​

Agent platform architecture is the set of shared services an organization uses to build, run, and govern AI agents. It typically covers an agent runtime, orchestration, task routing, memory, observability, identity, and policy enforcement.

What is an agent control plane?​

An agent control plane is the governance layer for an organization's agents. It handles inventory, identity, policy, tracing, and evaluation. It sits outside the request path and pushes policy to gateways that enforce it.

When should you use a multi-agent system instead of a single agent?​

Use multiple agents when a task spans domains with different data access rules or needs a different model per step. If one agent can handle the task reliably, keep it single, because it is easier to secure and debug.

When does a team need a dedicated agent platform?​

The usual triggers are operational. Several teams ship agents on different frameworks, finance needs costs by team, or prompts carry regulated data. Moving agents into production at scale is the other common trigger.