Circuit Breakers and Resilience Patterns for APIs

Every API eventually calls something that fails: a database that times out, a downstream service that slows down, a third-party dependency that goes dark during a deploy. What separates a resilient system from a brittle one isn't whether failures happen (they always do). It's whether one failure stays contained or spreads through every service that depended on it.

This guide covers the core resilience patterns for APIs: timeouts, retries with backoff, circuit breakers, and bulkheads, plus the supporting patterns of rate limiting and health checks. It also covers a question most explainers skip: which of these belong at the gateway, and which belong in application code.

What Is API Resilience?​

API resilience is an API's ability to keep functioning, in a degraded or limited form, when one of its dependencies fails, slows down, or becomes unreachable. A resilient API isn't one that never fails. It's one where a single dependency failure doesn't take down every service that calls it.

The mechanism behind most resilience failures is cascading failure. Say a payment service starts responding slowly because its database is under load. Every caller that isn't protected keeps holding open connections, waiting on threads, and retrying, which adds more load to an already struggling service. Within minutes, the slowdown in one component turns into an outage across several. Cascading failure is the reason a single misbehaving dependency can take an entire platform down, and it's the specific problem the patterns below exist to prevent.

In practice, this shows up as concrete engineering decisions: how long a gateway waits before giving up on a backend call, how many times a client retries a failed request and how it spaces those attempts out, and at what point a system stops calling a dependency altogether rather than keep hammering it while it's down.

Core API Resilience Patterns​

Four patterns form the backbone of API resilience. They address different failure modes, and in a mature system, they work together rather than in isolation.

Timeouts​

A timeout is a bound on how long a caller will wait for a response before treating the call as failed. Without one, a slow dependency doesn't just respond slowly: it holds a connection, a thread, or a memory buffer open indefinitely, and those held resources are what turn a slow dependency into an unavailable one for everyone else waiting behind it.

Setting a timeout is a trade-off, not a default value you copy from a tutorial. Too long, and the timeout does nothing to protect the caller during an incident. Too short, and normal, healthy responses under momentary load get treated as failures, which triggers retries that add more load. Most teams set timeouts per dependency, based on that dependency's own latency profile, rather than one blanket value across the whole system.

Retries with Backoff​

A retry re-sends a request that failed, on the assumption that the failure was transient (a dropped connection, a momentary blip) rather than a sign that the dependency is actually down. Backoff is the practice of waiting longer between each successive retry (often growing exponentially, with some randomness added as jitter) instead of retrying immediately.

Backoff matters because retrying immediately after a failure is close to the worst possible response: if a dependency is struggling because it's overloaded, every client retrying at the same instant adds a fresh burst of load at exactly the moment the dependency needs less of it. That coordinated pile-on is what's known as a retry storm, and it's a well-documented antipattern in production systems. Backoff, and capping the number of retry attempts, are the direct countermeasures.

Retries also compound across layers instead of adding up. If a client, a gateway, and an application service each retry independently, the deepest call in the chain gets attempted far more than any single layer's retry count suggests: a five-layer call chain that makes three attempts at every layer can multiply the load on the innermost dependency roughly 243x, not 15x, because each layer's retries are multiplied by every layer below it. The fix isn't retrying nowhere. It's picking one deliberate point in the stack to own retries (usually the layer closest to the failure), backed by a shared deadline or retry budget the whole chain respects, rather than letting every layer reason about retries in isolation.

Retries also need a boundary condition: a request should generally be retried only if it's known to be safe to repeat (idempotent), and only up to a small, fixed number of attempts, since retrying indefinitely just delays the failure and burns resources on both sides of the call.

Circuit Breakers​

A circuit breaker takes a different approach: instead of retrying a failing call, it stops attempting the call altogether once failures cross a threshold, and starts failing fast until the dependency shows signs of recovery. The pattern gets its name from the electrical circuit breaker it's modeled on. It was popularized by Michael Nygard's Release It! and is widely known through Martin Fowler's widely cited description of the mechanism.

A circuit breaker moves between three states. Closed is the normal state: requests pass through and the breaker just counts failures. Once failures cross a configured threshold, the breaker trips to open, and for a cooldown period it rejects calls immediately without attempting them: this is what "failing fast" means in practice. After the cooldown, the breaker moves to half-open and lets a small number of test requests through. If those succeed, it closes again, and if they don't, it reopens.

Circuit breakers and retries solve related but distinct problems. A retry assumes the operation will probably succeed if you try again. A circuit breaker assumes the operation is likely to fail and stops attempting it rather than trying again. The two are commonly combined: retry with backoff for the occasional blip, with a circuit breaker as the backstop that trips when the blips turn into a sustained outage.

Bulkheads​

A bulkhead isolates resources (connection pools, thread pools, or compute capacity) by workload or dependency, so that one failing or overloaded dependency can't exhaust resources needed by everything else. The name comes from ship design: a bulkhead is a partition that keeps a hull breach in one compartment from flooding the whole vessel.

The practical failure mode a bulkhead prevents is resource starvation from a single noisy dependency. If every call in a service shares one connection pool, and one downstream dependency slows down, that dependency alone can consume every available connection, starving calls to healthy dependencies that have nothing to do with the failure. Partitioning the pool per dependency contains the damage to the connections assigned to the failing one.

Where Each Pattern Lives: Gateway vs. Application Layer​

Most explainers describe these four patterns as if they all belong in the same place. In practice, some patterns are naturally centralized at the gateway, and others need to stay in application code, because they depend on knowledge only the application has. This is a starting framework, not an architectural rule: real systems split it based on what a given gateway supports and what a given team owns, and the placements below are a reasonable default rather than a strict boundary.

Patterns that fit naturally at the gateway: Rate limiting, circuit breaking on outbound calls to a given backend, and connection-level bulkhead isolation between backend pools all benefit from being enforced in one place rather than reimplemented per service. A gateway sees traffic to every backend it fronts, which puts it in a good position to notice when one backend is failing and to isolate the blast radius before it reaches every service behind it. Centralizing these at the gateway layer means a team defines the policy once instead of reimplementing it per service, but a new service doesn't get correctly configured protection just by sitting behind the gateway. Someone still has to attach the relevant policy, set thresholds that make sense for that specific backend, and confirm which downstream calls are actually protected. It's also worth being explicit about scope. This only covers traffic that passes through the gateway. Internal, service-to-service calls that bypass it need their own timeout, retry, and circuit-breaking logic.

Patterns that need to stay in application code: Retries with backoff and fine-grained timeout tuning often need context the gateway doesn't have. That context includes whether a particular request is safe to retry (idempotency is usually a business-logic property, not something visible at the transport layer), and how long a specific operation should reasonably take, which varies by endpoint far more granularly than a gateway-wide default can capture. This isn't a claim that timeouts belong exclusively in application code: gateways and service meshes commonly enforce a route- or upstream-level timeout as a coarse, always-on backstop, and some support connection-level retries as a similar safety net. What generally stays in the application is the fine-grained call: is this specific request safe to retry, and what's the right timeout for this specific operation, as opposed to the backend it happens to call?

A useful rule of thumb: patterns that are the same regardless of which service is calling (is this backend healthy, is this client over its rate limit) belong at the gateway. Patterns that depend on what a specific request means (is this particular write idempotent, how long should this particular query reasonably take) belong in the application.

Supporting Patterns: Rate Limiting and Health Checks​

Two more patterns don't prevent failure directly, but they reduce how often failure happens and how long it takes to detect.

Rate limiting: Caps how many requests a client or service can make in a given window. It doesn't recover from failure the way a circuit breaker does. Instead, it prevents a class of failure from happening in the first place, by stopping traffic from exceeding what a backend can handle before that backend starts timing out or erroring under load. Rate limiting is close to universally implemented at the gateway, since it's the first point that sees traffic from every consumer.

Health checks: Lets infrastructure (a gateway, a load balancer, or an orchestrator) know which backend instances are actually able to serve traffic, so it can route around instances that aren't. Three distinct checks get lumped under this name, and conflating them causes real incidents. A startup check confirms an instance has finished initializing. A liveness check confirms a running process hasn't hung or deadlocked. A readiness check confirms the instance can currently serve a request, which may include reaching its own dependencies. What matters most is what each check does on failure. A liveness check that also probes a remote dependency will restart a perfectly healthy process because something downstream is having a bad day, and if every instance shares that dependency, that's a synchronized restart loop layered on top of an outage, not a fix. A readiness check that strictly requires every dependency to be reachable has a milder but related problem. During a shared dependency outage, it can pull every instance out of rotation at once, which is the opposite of what you want when the instances themselves are fine. Neither "always check dependencies" nor "never check dependencies" is the right default for every check. Decide deliberately what each one should do on failure, and test that behavior, including a shared, partial dependency outage, rather than assuming a deep readiness check is automatically the safer choice.

Best Practices for Implementing Resilience Patterns​

A few practices consistently separate resilience patterns that hold up in production from ones that look good in a design doc:

• Pair retry with circuit breaking, don't rely on retry alone: Retrying without a circuit breaker as a backstop means a sustained outage just gets retried into a worse outage.
• Set timeouts per dependency, not one global default: A cache lookup and a report-generation call don't have the same acceptable latency, and a single timeout value tuned for one will be wrong for the other.
• Make retries idempotency-aware: Automatically retrying a non-idempotent write (like a payment charge) on a timeout can duplicate the operation if the original request actually succeeded and only the response was lost.
• Monitor breaker state and fallback rate, not just uptime: A circuit breaker that's open half the time is telling you something about a dependency's health that a simple uptime metric won't surface.
• Test the failure path, not just the happy path: Run a deliberate failure exercise (kill a dependency or inject latency) and confirm the specific things that are supposed to happen: the timeout budget is actually respected, retries stop at the configured count instead of compounding across layers, the circuit breaker actually opens and later half-opens and recovers, and a failing pool stays isolated instead of starving healthy ones. A circuit breaker or fallback that's never been exercised against a real failure is unverified code, regardless of how correct the configuration looks on paper.
• Centralize what's genuinely shared, and leave the rest in application code: Pushing every resilience decision to the gateway just to have "one place" for it tends to produce gateway configuration that's too coarse for any single service's actual needs.

How WSO2 API Gateway Implements Resilience Patterns​

The patterns above are architectural decisions before they're configuration, but the "where each lives" framework from earlier in this guide has a direct, practical conclusion: the patterns that belong at the gateway layer are best enforced centrally, rather than rebuilt inside every service.

That's the role the WSO2 API Gateway layer plays for inbound API traffic. Rate limiting is available as a gateway policy, and the gateway's resilience configuration covers timeouts: a per-API or per-operation request timeout and idle timeout, plus a per-upstream connect timeout, all overriding gateway-wide defaults. That keeps a slow or unreachable backend from holding requests open indefinitely, without every team writing and maintaining its own timeout handling. A service put behind the gateway gets these protections once the relevant policies and timeout values are actually set for that backend. Placing a service behind the gateway doesn't configure it automatically, and calls between internal services that bypass the gateway aren't covered by it at all.

It's worth being precise about scope here, because it's a common point of confusion: this is about inbound traffic from clients to your services. As more applications also make outbound calls to LLM providers, that's a related but architecturally distinct traffic path, one governed separately by the AI Gateway, not as a mandatory hop every API request passes through (more on how the two divide traffic). On that path, the same resilience thinking applies in a different shape: an LLM fallback is a routing decision, rerouting to a different model or provider, rather than a circuit breaker's stateful trip-and-cooldown behavior. Mature setups typically use both together: retry with backoff for a brief blip, a circuit breaker to stop hammering a provider that's stayed down, and fallback to route around it while the breaker is open. Providers aren't drop-in equivalents of each other just because they're both LLMs. A fallback that swaps providers also has to reconcile differences in request and response format, tool-calling behavior, safety controls, and where the data is allowed to go, or the fallback path can become its own source of failures. The two gateways address separate traffic directions under one control plane, rather than one queuing behind the other.

Whether that outbound path needs its own dedicated gateway layer is a question worth asking honestly rather than assuming yes by default. The signal isn't a volume figure. It's whether you're routing across more than one model provider, whether you need to attribute AI spend across teams, whether prompts touch regulated data that needs an audit trail, or whether the AI feature has moved from a prototype into something production traffic depends on. A single application calling a single provider in an early prototype doesn't need this layer yet. A production feature spanning providers with real cost and compliance stakes generally does.

Conclusion​

Timeouts, retries with backoff, circuit breakers, and bulkheads each address a different way a dependency can fail, and rate limiting and health checks reduce how often failure happens and how fast it's detected. None of them is a substitute for the others, and none of them needs to live in the same place. Patterns that generalize across services belong at the gateway. Patterns that depend on what a specific request means belong in application code. Getting that split right is most of what separates a resilience strategy that holds up under a real incident from one that only looked complete in a design review.

Frequently Asked Questions​

What's the difference between a retry and a circuit breaker?​

A retry re-attempts the same call on the assumption it will probably succeed next time. A circuit breaker stops attempting the call once failures cross a threshold, on the assumption further attempts will likely fail too.

Do circuit breakers replace retries?​

No. They solve different problems and are typically combined: retry with backoff for brief, transient errors, with a circuit breaker as the backstop for sustained failures.

Should resilience patterns live at the gateway or in application code?​

It depends on the pattern. Rate limiting, backend circuit breaking, and connection-pool isolation generalize well at the gateway. Request-level timeout tuning and idempotency-aware retries usually need context only the application has.

What causes a cascading failure?​

An unprotected dependency failure (one without a timeout, retry limit, or circuit breaker) consumes shared resources like threads or connections faster than the system can free them, and that resource exhaustion spreads to services that depend on the affected one.

Is a health check enough to detect a failing backend?​

A liveness check alone isn't, since it only confirms a process is running. A readiness check that confirms the service can reach its dependencies gives a stronger routing signal, but checking every dependency strictly can pull all instances out of rotation during a shared outage, so the right scope depends on testing the specific failure behavior you want, not on checking as much as possible by default.