API Gateway Architecture: How API Gateways Actually Work

Every microservices system eventually hits the same problem. Clients need to talk to dozens of backend services, and none of those services should have to handle authentication, rate limiting, and routing on their own. API gateway architecture solves this by putting a single, purpose-built component in front of the service fleet.

This guide breaks down how that architecture actually works, where it fits within the broader patterns covered in API Architecture: Patterns, Principles, and Modern Approaches, and where it starts to strain at scale.

What Is an API Gateway?​

An API gateway is a single entry point that sits between clients and a set of backend services. It receives every incoming request, applies cross-cutting policies like authentication and rate limiting, and routes the request to the right service.

For a request that maps to a single backend, the gateway simply returns that service's response. When a request needs data from more than one service, the gateway can compose those results into a single response, but that's the exception rather than the default. Instead of every service reimplementing the same plumbing, the gateway handles it once, centrally.

How API Gateway Architecture Works​

At a structural level, an API gateway architecture looks less like a single box and more like a facade. It defines the public interface of the application while hiding the internal service topology from callers. Chris Richardson's microservices pattern catalog describes the gateway as the single entry point for all clients. It's a widely used reference point rather than the pattern's original source, and the framing still holds.

Clients send requests to one stable address. The gateway decides what happens next: which service handles the request, which policies apply before it gets there, and what shape the response takes on the way back.

That single entry point is also what makes the rest of the architecture manageable. A service fleet can grow, be renamed, be redeployed to new infrastructure, or be split into smaller services, and none of that has to be visible to the clients calling through the gateway. The gateway's own configuration absorbs that churn, instead of every client having to track it.

Request Routing and Reverse Proxying​

The gateway's first job is routing. It works as a reverse proxy, accepting inbound calls and forwarding them to the appropriate backend service based on the URL path, HTTP method, or request headers. A mobile app might call /orders, and the gateway maps that path to an internal order service without the client ever knowing that service's actual network location.

This is also where load balancing happens. If a service runs multiple instances, the gateway distributes traffic across them so no single instance gets overwhelmed.

Routing at this layer is what makes a single entry point workable in practice. Without it, every client would need to track the location, version, and health of every backend service directly, which becomes unmanageable once a system grows past a handful of services.

Protocol Translation​

Backend services rarely all speak the same protocol. One team's service might expose gRPC, another might still run SOAP, and a newer service might be event-driven. Protocol translation lets the gateway present a single, consistent interface to clients, typically REST or GraphQL, while translating requests into the protocol a given backend expects.

That works only for the protocol pairs the gateway has been configured to bridge. Not every combination is automatic: each translation needs its own adapter and field mapping, set up ahead of time for that backend. A pair without one doesn't get translated just because the gateway supports translation in general.

Cross-Cutting Concerns (Auth, Rate Limiting, Logging, Caching)​

This is where most of an API gateway's operational value shows up.

• Authentication: Rather than every backend service implementing its own perimeter authentication, the gateway validates API keys, JWTs, or OAuth tokens once at the edge and rejects invalid or unauthenticated requests before they reach a service. Passing that check doesn't mean a caller can touch every resource behind it, so services still run their own object-level authorization, checking what this specific caller can access, using the identity the gateway propagates rather than re-authenticating from scratch. Backend services also typically aren't reachable directly, which is what actually stops a caller from bypassing the gateway's checks altogether.
• Rate limiting and throttling: These protect backend capacity by capping how many requests a given client or API key can make in a time window.
• Centralized logging: Operators get one place to see request volume, latency, and error rates across the entire API surface, instead of stitching together logs from dozens of services.
• Response caching: Applied at the gateway, caching can serve repeat requests without hitting a backend service at all, cutting both latency and load.

Handled well, these cross-cutting concerns are the real argument for API gateway architecture. Handled poorly, with too much business logic piled into gateway configuration, they turn the gateway into a de facto enterprise service bus, which defeats the purpose of keeping services independently deployable.

The API Gateway Pattern vs. Direct Client-to-Service Communication​

The alternative to a gateway is direct client-to-service communication, where clients call backend services individually. This works fine at small scale. It stops working once a single client screen needs data assembled from several services, because the client itself ends up managing multiple connections, handling partial failures, and aggregating responses that a gateway could have composed on the server side.

Direct communication also duplicates every cross-cutting concern. Authentication, rate limiting, and request validation get implemented separately inside each service instead of once at the edge, which means inconsistent enforcement and more code to audit.

The API gateway pattern trades that duplication for a single control point. The cost is a new component the whole system now depends on, a tradeoff covered in the challenges below.

Backends for Frontends (BFF): A Gateway Variation​

Backends for Frontends is a variation on the gateway pattern rather than a replacement for it. Instead of one general-purpose gateway serving every client type, a BFF architecture gives each client (web, mobile, or a partner integration) its own dedicated backend, tailored to that client's specific data and formatting needs.

That backend doesn't have to be a full standalone gateway. It's often an application-level service that sits behind the shared API gateway rather than replacing it, taking on only the shaping and routing logic specific to its client. A mobile BFF might return a trimmed payload optimized for a small screen and limited bandwidth, while a web BFF composes a richer response from the same underlying services.

Teams reach for BFF when client needs diverge enough that a single shared gateway configuration starts accumulating client-specific branches and exceptions.

API Gateway vs. Service Mesh​

API gateways and service meshes are often confused, because both apply policy to traffic and both can enforce authentication or routing rules. The real distinction is which traffic each one governs, not a fixed order every request has to follow.

• An API gateway sits at the edge and governs north-south traffic: requests coming in from external clients to the services behind it.
• A service mesh operates inside the cluster and governs east-west traffic: the service-to-service calls that happen after a request has already entered the system. It typically works through a sidecar proxy deployed alongside each service instance, with a separate control plane handling configuration rather than the traffic itself.

A single transaction can still touch both in sequence, but an incoming client request doesn't need to cross both layers to be valid. It enters through the API gateway, and if the service it lands on then needs to call two other internal services to fulfill it, those follow-on calls are what a service mesh secures and observes, independently of the gateway.

Teams running fewer than a handful of services can usually skip a mesh entirely and rely on the gateway plus ordinary retry logic. A mesh earns its complexity once service-to-service traffic, and the mTLS, retries, and tracing it needs, becomes too large to reason about by hand.

Common API Gateway Architecture Challenges​

API gateway architecture solves real problems, but it introduces its own tradeoffs.

• Single point of failure: Because every client request passes through the gateway, an outage there affects the entire system, not just one service. Redundant gateway instances behind a load balancer and proper health checks make this risk manageable, not eliminated, and it has to be designed for deliberately rather than assumed away.
• Added latency: The gateway is an extra network hop. In most deployments that overhead is small, but it grows with the amount of work the gateway does per request, particularly transformation, aggregation, or heavier authentication checks.
• Operational complexity: The gateway is another piece of infrastructure to run, version, and monitor. Left unchecked, teams tend to push more and more routing and business logic into gateway configuration over time. That can turn a lightweight edge component into a de facto enterprise service bus that every team has to coordinate changes through.

None of these challenges argues against using a gateway. They argue for treating it as a first-class piece of infrastructure, with its own redundancy plan and its own limits on scope, rather than a place to accumulate logic that belongs in services.

Modern API Gateway Architecture: Multi-Gateway and AI Traffic​

Multi-Gateway Estates​

Much of what's written about API gateway architecture assumes a single-gateway deployment: pick one gateway product and standardize on it. In practice, larger organizations rarely end up that tidy. Different teams land on Kong, a cloud provider's native gateway, or a self-hosted option for reasons that made sense at the time. Re-platforming all of them onto one gateway is rarely worth the disruption. It usually means months of migration work to solve a governance problem that doesn't actually require throwing out working infrastructure.

WSO2's API Platform approaches this from the other direction, by separating configuration from runtime enforcement. Its API Control Plane is where policies, access controls, and monitoring are defined, and it pushes that configuration out to the platform's own API Gateway component. The third-party gateways it federates (AWS API Gateway, Azure API Management, Kong, and Envoy) keep running and enforcing their own traffic, with no migration to one vendor's runtime required. The control plane discovers their APIs, applies the same governance compliance policies, and publishes them to one developer portal, so the whole fleet is visible and governed from one place.

That shared visibility and governance is the practical difference between "multi-gateway" as a marketing claim and multi-gateway as an operational reality.

AI Traffic​

Gateway architecture is also extending beyond standard REST traffic. As applications add generative AI features, they introduce a second, separate traffic path: outbound calls from an application to an LLM provider, alongside whatever inbound client traffic the API gateway already governs. That outbound path carries its own concerns, such as token-based cost tracking, multi-model routing, and prompt-level guardrails, which a standard API gateway was never built to inspect.

The same architecture is starting to account for a third path too: AI agents calling internal tools and APIs directly. That path needs its own authentication and throttling, rather than being folded into either of the other two.

These are different traffic types with different control points, not one pipeline every request has to complete. They can still occur in sequence within a single transaction. A client's request comes in through the API gateway, and if handling it means the service needs to call a model provider, that outbound call goes through the AI Gateway next, as a second hop in the same transaction. An agent calling an internal tool is a separate path again, with its own auth requirements, and it doesn't depend on either of the other two having happened first.

What changes between a single-gateway setup and this one is whether all three paths are governed by one policy engine or by three disconnected ones. Both the outbound LLM path and the inbound agent-to-tool path are handled by the same AI Gateway component, not by two separate gateway products. A team adding AI features is extending one part of the WSO2 API Platform rather than standing up a disconnected second stack with its own identity and policy model.

The trigger for adding this layer is architectural, not a token-count threshold. Teams that route to multiple model providers, need to attribute AI spend to specific teams, handle regulated data inside prompts, or are moving an AI feature from prototype to production scale tend to need dedicated AI traffic governance. A single team calling one provider's SDK directly from a prototype usually doesn't, yet.

Conclusion​

API gateway architecture comes down to one design decision: concentrate cross-cutting concerns like routing, authentication, and rate limiting in a single edge component instead of duplicating them across every service.

That decision brings real tradeoffs: a new single point of failure to manage, a new hop of latency, and a new piece of infrastructure to operate. For most systems fronting multiple backend services, those tradeoffs are usually worth it.

Explore WSO2's API Platform to see how its API Control Plane can manage that architecture across multiple gateway technologies at once.

Frequently Asked Questions​

What is the difference between an API and an API gateway?​

An API is the interface a service exposes so other software can call it. An API gateway is a separate component that sits in front of many APIs, applying routing, authentication, and rate limiting before a request ever reaches the service behind it.

What are the disadvantages of using an API gateway?​

The main tradeoffs are a new single point of failure, an added network hop that introduces latency, and extra operational overhead from running and maintaining another piece of infrastructure. All three are manageable with redundancy and clear scope, but not automatically eliminated.

How do I design an API gateway?​

Start by deciding which cross-cutting concerns (authentication, rate limiting, routing, and caching) move to the gateway and which stay in services. Then plan for redundancy from the start, since the gateway becomes a critical dependency the moment services rely on it.

Can you give me an example of an API gateway?​

A common example is a single gateway in front of an e-commerce system's order, inventory, and payment services. It handles auth and rate limiting for all three, so a mobile app only needs to call one endpoint instead of three.