Service Mesh vs API Gateway: When You Need Each (or Both)

An API gateway and a service mesh both sit in the traffic path of a modern microservices application, and both get called "the thing that manages your traffic." That overlap is exactly why the two get confused. They don't compete for the same job. They govern two different directions of traffic, and most production systems that have grown past a handful of services end up needing some mix of both.

This guide breaks down what each one actually does, why the north-south vs. east-west distinction matters more than any feature checklist, and how to decide whether you need an API gateway, a service mesh, or both running side by side. For the broader architectural context this decision sits inside, see API Architecture: Patterns, Principles, and Modern Approaches.

Service Mesh vs. API Gateway: The Core Difference​

An API gateway manages north-south traffic: requests moving between external clients and your services. A service mesh manages east-west traffic: requests moving between your services themselves.

That single distinction explains almost every other difference between them. An API gateway is the front door. It sits at the edge of your architecture, and every external client, whether that's a mobile app, a partner integration, or a browser, talks to your system through it. A service mesh lives entirely inside the perimeter. Once a request has been authenticated and routed in, a service mesh takes over the job of getting it from one internal service to the next, securely and reliably, without any of the calling services needing to know how.

Neither tool was built to do the other's job, and neither replaces the other, even though the split isn't absolute. Gateways aren't only external-facing: many organizations run internal API gateways to govern traffic between internal consumers and internal services. A service mesh, in turn, typically ships its own ingress and egress components to manage traffic entering or leaving the mesh. So north-south and east-west describe each tool's primary use, not a strict boundary neither ever crosses. The more useful distinction is what each is built to own. An API gateway is built around external API-consumer identity, versioning, and product lifecycle. A service mesh is built around service identity, service-to-service policy, and the mechanics of one workload calling another.

North-South vs. East-West Traffic: Why This Distinction Matters​

The terms "north-south" and "east-west" come from network diagrams, where external traffic is conventionally drawn flowing top to bottom and internal traffic side to side. NIST's guidance on securing microservices architectures, SP 800-204A, uses this exact split. Traffic between clients and an application service is north-south, and traffic between one microservice and another is east-west.

The distinction matters because the two traffic types have almost nothing in common operationally:

• Volume and predictability: North-south traffic is bursty and driven by external demand. East-west traffic is typically much higher in volume (one external request can fan out into a dozen internal calls) and follows patterns your own services define.
• Trust model: A client hitting your API from the internet is untrusted by default. A service calling another service inside your own cluster is a different trust problem entirely, closer to "which of my own workloads is allowed to talk to which."
• Failure modes: A gateway failure is visible immediately. Every external request breaks. A mesh-level failure (a slow downstream service, a broken retry policy) can hide inside a request for a while before it surfaces as a timeout at the edge.
• What you need to observe: Gateway metrics center on requests per second, latency percentiles, and error rates at the edge. Mesh metrics center on service-to-service latency, retry rates, and circuit-breaker trips between specific pairs of services.

Architecturally, a single external request commonly crosses both boundaries: it enters through the gateway, which authenticates and routes it to a front-line service, and from there it can fan out into any number of service-to-service calls that the mesh governs separately. The two layers aren't a single pipeline with the mesh as simply "the next hop after the gateway," though. Each is an independent enforcement point applying controls suited to its own boundary. The gateway's policies don't extend into the mesh's east-west traffic, and the mesh's policies don't extend out to the gateway's north-south traffic.

What an API Gateway Does​

An API gateway is the centralized entry point for every request coming in from outside your system. Instead of exposing dozens of internal services directly to the internet, clients talk to one stable endpoint, and the gateway handles the concerns that would otherwise need to be duplicated in every backend service:

• Authentication and authorization: Validating API keys, JWTs, or OAuth tokens before a request is allowed through.
• Rate limiting and throttling: Protecting backend services from being overwhelmed by a single noisy consumer.
• Load balancing: Spreading incoming requests across healthy service instances.
• Request routing and transformation: Directing traffic based on path, method, or header, and translating between client-facing and internal payload formats.
• Circuit breaking at the edge: Failing fast and returning a controlled error instead of letting a struggling backend cascade into a full outage for external callers.

Because it sits at the perimeter, an API gateway is also the natural place to enforce API-level governance: versioning, developer-facing documentation, and usage policy for whoever is consuming your APIs, whether that's a partner, a mobile client, or (increasingly) an AI agent calling a tool over MCP. When that agent originates outside your organization, that call is still north-south traffic managed at the gateway, the same as any other external consumer, even though the caller is a model rather than a person clicking a button. An agent you're running internally, calling one of your own tools, is a different case, closer to the east-west traffic covered next.

What a Service Mesh Does​

A service mesh is a dedicated infrastructure layer for east-west traffic. Rather than building retry logic, mutual TLS, and service discovery into every microservice's code, a mesh handles these concerns at the infrastructure level, typically through a proxy that sits alongside each service.

The traditional approach, still how Linkerd and most Istio deployments run today, injects a sidecar proxy next to every workload. Istio's sidecar proxy is Envoy. Linkerd uses its own purpose-built proxy, linkerd2-proxy, written in Rust, rather than Envoy. Either way, the sidecar intercepts all inbound and outbound traffic for that service and handles:

• Service discovery, so services can find each other without hardcoded addresses.
• Mutual TLS (mTLS), encrypting and authenticating traffic between services automatically, which is also the piece of infrastructure most zero trust security models for microservices lean on.
• Load balancing and traffic shifting, supporting canary releases and gradual rollouts between service versions.
• Circuit breakers and retries, isolating failing services so one slow dependency doesn't take down its callers.
• Fine-grained observability, tracing and metrics for every service-to-service call, not just the ones that touch the edge.

A newer generation of meshes has started moving away from the sidecar model. Istio's "ambient mode" replaces the per-pod sidecar with a shared, per-node proxy (ztunnel) that handles mTLS and identity at Layer 4. It adds an optional waypoint proxy only where a service needs Layer 7 policy like HTTP-aware routing. A waypoint is typically scoped to a namespace by default, but it can be configured per service or per workload for more granular control. See Istio's waypoint documentation for the full set of scopes. That's a meaningful shift for teams who found the operational overhead of a sidecar in every pod, one more container to patch, monitor, and account for at scale, hard to justify when most services only need basic mTLS and don't need L7 traffic shaping.

Key Differences at a Glance​

Dimension	API Gateway	Service Mesh
Traffic direction	North-south (client to service)	East-west (service to service)
Position in architecture	Edge of the system, perimeter	Inside the cluster, between every service
Observability focus	Request throughput, latency, error rate at the edge	Per-service-pair latency, retries, circuit-breaker trips
Deployment and management	Centralized, one (or a few) gateway instances	Distributed, a proxy alongside (or a shared layer near) every service
Fault tolerance role	Protects backends from external overload	Isolates failing services from each other internally

Reading this table as two different toolsets built for two different operational problems is more useful than reading it as a scorecard. Neither is a strictly "better" version of the other. They were built to sit in different places.

When You Need Just an API Gateway​

Most systems need an API gateway well before they need a service mesh. Once you're exposing any services to external clients, an API gateway is worth adopting almost by default. The decision isn't really about how many services you run, but about not wanting to reimplement authentication, rate limiting, and routing separately in every one of them.

An API gateway alone is usually enough when:

• You have a moderate number of backend services, and service-to-service calls are simple, low-volume, or largely absent (a monolith-with-a-gateway-in-front pattern, or a small number of services calling each other directly).
• Your operational risk is concentrated at the edge, such as bad actors, traffic spikes, or malformed client requests, rather than complex internal failure cascades.
• Your team doesn't yet have the platform capacity to run and maintain a separate mesh control plane and data plane on top of everything else.

When You Need Just a Service Mesh​

A service mesh with no API gateway in front of it is a less common pattern, and it usually only makes sense for internal-only systems: a platform where every caller is another internal service, and there's no direct external client traffic to govern at all. In that scenario, all your traffic is genuinely east-west, and the concerns a gateway would solve (external authentication, public rate limiting) simply don't apply.

A service mesh becomes clearly worth adopting on its own (or ahead of a gateway rework) when:

• Your service-to-service traffic has grown complex enough that debugging a failure means tracing a request across five or six internal hops, and you have no consistent way to do that today.
• You need mTLS and consistent authorization between services for compliance or zero-trust reasons, and doing that in application code, service by service, has become unmanageable.
• You're running canary releases or gradual rollouts between service versions and want that traffic-shifting logic out of application code and into infrastructure.

When You Need Both, and How They Work Together​

Most production systems past a certain scale end up running both. The two are complementary specifically because they don't overlap. An API gateway governs what comes in from outside, and a service mesh governs what happens once it's inside.

A typical coexistence pattern looks like this. An external request hits the API gateway, which authenticates the caller, applies rate limits, and routes the request to the appropriate front-line service. From there, that service may call several other internal services to fulfill the request, calls that never touch the gateway again and are instead handled entirely by the mesh, which applies mTLS, retries, and circuit breaking between those internal hops.

The two layers stay decoupled deliberately. The gateway doesn't need to understand the internal service graph, and the mesh doesn't need to understand external API contracts or developer-facing versioning. Each layer can evolve, scale, and fail independently of the other. It's not a single pipeline where a request passes through the gateway and then through the mesh as a second sequential hop, but two separate governance layers each doing their own job over the same request's lifecycle.

Istio vs. API Gateway: Clearing Up a Common Confusion​

Istio is a service mesh, not an API gateway, but the confusion around this is understandable for a couple of reasons.

First, Istio does ship an ingress gateway component, which handles north-south traffic entering the mesh. That component genuinely does gateway-shaped work such as TLS termination, basic routing, and edge-facing policy. Stacked up against a bare runtime API gateway (just the traffic-handling proxy, with no portal or lifecycle layer), that's a fair, close comparison. Measured against a full API management platform, it's a different comparison. Developer portals, API product management, and full lifecycle tooling are capabilities some API gateway products bundle on top of the runtime proxy, not something every gateway, or Istio's ingress component, provides by definition.

Second, the rise of ambient mode has blurred the line further at the infrastructure level. Kubernetes' own Gateway API, originally built purely for north-south ingress routing, has extended (through its GAMMA initiative) to also configure east-west mesh traffic using the same resource types. That means the same API surface can, in principle, describe both an ingress route and a mesh route, which makes "is this gateway config or mesh config" a less useful question than it used to be.

None of that changes the underlying answer. Istio's core job is still east-west traffic between your services. If you need a full-featured API gateway (front door authentication, API productization, developer portals, rate limiting policies tailored to external consumers), Istio's ingress component isn't a substitute for one, even though it technically sits at the edge too.

A Strong Fit for Architects: How WSO2 Supports Both Layers​

AI traffic introduces boundaries of its own. They aren't a direct match for north-south and east-west, but the same underlying idea, that different traffic directions cross different trust boundaries and need governance suited to each, still applies. An outbound call from your application to an external LLM provider crosses your perimeter outward, the opposite direction from a north-south client request but a boundary crossing all the same. An AI agent calling one of your tools over MCP can originate from outside your organization or from an agent you're running internally, so it isn't automatically "inbound" in the way a north-south client request is. What matters is which specific boundary that call crosses, not a mechanical label.

That same governance model extends to AI traffic. WSO2's AI Gateway is built to govern both of these AI boundaries under one component. It applies controls to outbound calls from your applications to LLM providers, and to calls from AI agents invoking your tools over MCP, rather than requiring two disconnected policy sets. For an architect already separating client traffic from service traffic, that's a familiar instinct extended to a newer traffic category: govern each boundary with the policies suited to what's actually crossing it, rather than mapping it mechanically onto north-south and east-west.

For teams managing gateways across multiple clouds or vendors, WSO2's API Control Plane manages WSO2's own gateways alongside federated third-party gateways, including Kong, AWS API Gateway, Azure API Management, and Envoy. It discovers their APIs, applies governance compliance policies, and surfaces them in one developer portal, without rerouting the underlying traffic, rather than requiring separate tooling per gateway technology. That's the same coexistence principle that applies to a service mesh sitting behind any of those gateways. The control plane governs the perimeter across your gateway fleet. It doesn't attempt to manage what a service mesh is doing behind it. For a deeper comparison of how WSO2 separates API and AI traffic governance, see AI Gateway vs API Gateway: Key Differences.

Conclusion​

An API gateway and a service mesh solve different problems because they govern different traffic. One manages what comes in from outside. The other manages what happens between your own services once a request is inside. Most systems start with a gateway, add a mesh once internal traffic gets complex enough to justify it, and run both side by side rather than choosing one over the other permanently.

If you're evaluating how to govern API, AI, and agent traffic under one consistent model rather than stitching together separate tools for each, see how the WSO2 API Platform approaches API and AI gateway governance together.

Frequently Asked Questions​

Can Istio replace an API gateway?​

Istio's ingress gateway component handles basic north-south routing into the mesh, comparable to a bare-bones API gateway, but it doesn't provide the API productization, developer portal, and lifecycle management that a full API management platform typically adds on top of a gateway.

Do I need both a service mesh and an API gateway?​

Most systems need an API gateway well before they need a service mesh. You'll typically add a service mesh once service-to-service traffic and failure modes become complex enough that managing them in application code, one service at a time, stops scaling.

What is the main difference between an API gateway and a service mesh?​

An API gateway manages north-south traffic between external clients and your services. A service mesh manages east-west traffic between your services themselves. They govern different traffic directions rather than competing for the same job.

What are the downsides of a service mesh?​

A service mesh adds operational complexity: a control plane to run, proxies to monitor (whether per-pod sidecars or a shared per-node layer), and a new set of failure modes to understand. Teams often adopt it once internal traffic complexity justifies that overhead, rather than by default.