API Rate Limiting: How It Works and Best Practices

A single misbehaving script, a retry loop with no backoff, or one client that polls your API every second instead of every minute can do more damage to a backend than a genuine traffic spike. Rate limiting is the control that stops any one client from taking the service down for everyone else.

It sounds simple, but it gets more nuanced as soon as you have to decide what counts as a request, which unit of time to measure it in, and what to do when a client's traffic looks less like requests and more like tokens.

What Is API Rate Limiting?​

API rate limiting is the practice of capping how many requests a client can send to an API within a defined time window, and rejecting or delaying requests that exceed that cap. It protects backend capacity, keeps response times predictable for every consumer, and gives an API owner a way to enforce fair use across clients with very different traffic patterns.

Rate limiting vs. throttling​

The two terms are often used interchangeably, and in casual conversation that's fine, but they describe slightly different things. Rate limiting is the policy: a defined number of requests per client per time window. Throttling is what happens at enforcement time, when the gateway slows down, queues, or rejects requests that would push a client past that policy. Most platforms bundle both into a single "rate limiting" configuration, but the distinction matters when you're debugging why a client is being rejected instead of queued, or the other way round.

Why API Rate Limiting Matters​

Every API request costs something: CPU cycles, database connections, calls to a backend service, and sometimes real money if that backend is a third-party service billed per call. Without a cap, a single client, whether malicious or simply misconfigured, can consume a disproportionate share of that capacity and degrade the experience for everyone else.

OWASP names this risk directly. API4:2023, Unrestricted Resource Consumption, covers exactly this failure: an API that doesn't limit how much network bandwidth, CPU, memory, or storage a client can consume is exposed to both denial-of-service conditions and unplanned operational cost. Rate limiting is the primary control for this risk, alongside request size limits and pagination caps.

There's also a fairness dimension that's easy to miss. A free-tier client polling aggressively can crowd out a paying customer making occasional, legitimate calls if there's no per-client accounting. Rate limiting applied per API key, per IP address, or per authenticated identity is what turns "whoever asks first wins" into a policy you actually control.

Security teams also treat rate limiting as a first line of defense against credential stuffing and brute-force login attempts, since both depend on sending far more requests than a legitimate user ever would in the same window. It won't stop a determined, distributed attack on its own, but it raises the cost of the attempt and buys time for other controls, such as account lockouts and anomaly detection, to kick in.

How API Rate Limiting Works​

At its core, a rate limiter tracks a client's consumption against a configured budget. Depending on the algorithm, that budget is either a request count inside a fixed or rolling time window, or a token allowance that refills and drains continuously. The tracking key is usually an API key, the subject claim of an OAuth token, or the client's IP address when neither of those is available. Once a client exceeds its budget, the gateway or service rejects further requests until the count resets or the allowance refills.

Rate limit response headers​

Well-behaved APIs tell clients where they stand before they get rejected. The convention most developers recognize is a set of X-RateLimit-* headers (X-RateLimit-Limit, X-RateLimit-Remaining, and often X-RateLimit-Reset) returned on every response, so a client can slow itself down before it hits a hard wall.

These header names aren't an official standard. They're a convention that started with early API providers and was copied widely enough to become a de facto expectation. The IETF's HTTPAPI working group is standardizing this area separately, and its current draft defines two structured fields, RateLimit and RateLimit-Policy, rather than the X--prefixed names most APIs use today. Until that draft becomes an RFC, expect the X-RateLimit-* convention to remain what you see in practice.

A typical response carrying rate-limit information looks like this:

HTTP/1.1 200 OK
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 842
X-RateLimit-Reset: 1735689600

What happens when a rate limit is exceeded​

When a client goes over its limit, the server should respond with HTTP status code 429 Too Many Requests, which the IETF defined for exactly this situation. The 429 response can include a Retry-After header telling the client how long to wait before trying again, which is the detail a rate-limited client most needs in order to back off properly instead of hammering the endpoint in a retry loop.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 30

{
  "error": "rate_limit_exceeded",
  "message": "Too many requests. Retry after 30 seconds."
}

A client that ignores Retry-After and retries immediately is the most common reason a rate-limited client ends up with worse results rather than better ones. The standard client-side fix is exponential backoff with jitter: waiting a randomized, increasing interval between retries.

Common API Rate Limiting Algorithms​

There's no single correct way to count "requests in a time window." Four algorithms cover almost every production rate limiter you'll come across, and each makes a different trade-off between accuracy, memory cost, and how it handles bursts of traffic.

Token bucket is a common choice for general-purpose API gateways, though not a universal default. It naturally supports "a steady rate most of the time, with the occasional small burst," which matches how real client traffic behaves more closely than a hard per-window cap. A simplified version looks like this:

def allow_request(bucket):
    now = current_time()
    elapsed = now - bucket.last_refill
    bucket.tokens = min(bucket.capacity, bucket.tokens + elapsed * bucket.refill_rate)
    bucket.last_refill = now

    if bucket.tokens >= 1:
        bucket.tokens -= 1
        return True
    return False

To make that concrete, take a bucket with a capacity of 10 tokens and a refill rate of 1 token per second. A client that has been idle can burst up to 10 requests immediately, then settles into a steady 1 request per second. If it fires 15 requests all at once, 10 are admitted and 5 are rejected with a 429. One second later, the bucket has refilled by exactly one token: enough for a single follow-up request, not another burst. Getting a full burst of 10 again requires the bucket to sit idle for the full 10 seconds it takes to refill.

It's worth being clear that the pseudocode above is a single-process illustration. The read, refill, check, and decrement steps have to happen atomically for a given limiting key, or two concurrent requests can both read the same token count and both be admitted when only one should be.

In a deployment with several gateway replicas, each keeping its own independent in-memory bucket, the effective global limit becomes the advertised limit multiplied by the number of replicas. Production implementations usually solve this with a shared, atomic counter, such as a Redis-backed token bucket that uses atomic operations, or by splitting the budget across replicas and accepting a coarser approximation of the global limit. Either way, you also need a stated failure policy: whether to fail open (admit requests) or fail closed (reject them) if the shared state becomes unavailable. That's a decision to make deliberately, not by default.

GitHub's REST API is a useful real-world reference. Every response carries x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset headers, and the primary limit resets at a fixed point in time rather than continuously. That's the fixed-window pattern described above, tuned to GitHub's own traffic and authentication tiers, not a template every API should copy. Those headers describe GitHub's primary rate limit only. GitHub also enforces separate secondary rate limits for rapid concurrent requests and other abuse patterns, which follow different rules and aren't reflected in the same headers. Check GitHub's rate limit documentation directly before assuming the primary-limit headers tell the whole story.

None of the four algorithms is universally right. The best choice depends on whether occasional bursts are acceptable or genuinely harmful to your backend, and whether the smoother enforcement of finer-grained tracking is worth the extra state.

Where to Enforce Rate Limiting​

Rate limiting can sit at more than one point in a request's path. A single request can pass through all of them in turn (the edge, then the service, then an outbound call), but each stop enforces its own budget, with a different key, a different unit, and a different owner. Clearing one checkpoint says nothing about whether the request will clear the next.

At the client-facing edge (API gateway or proxy). This is where most per-client, per-key rate limiting happens. It's the earliest point at which you know who's calling, and rejecting an over-limit request here means it never reaches your backend. It's also where X-RateLimit-* headers and 429 responses usually come from. The API gateway layer of a platform is the natural place to configure this for REST, GraphQL, and other synchronous traffic.

At the individual service. Some teams add a second, coarser limit inside the service itself, independent of the gateway's limit. It acts as a backstop in case traffic reaches the service by a path that bypasses the gateway, or because the service has its own resource ceiling that the gateway doesn't know about.

On outbound calls to dependencies. If the same request triggers a call to a third-party service, a database, or an LLM provider, that outbound call is subject to its own limit, set by the dependency and tracked against its own key and window. A request can clear the edge and service limits and still be throttled here. And a single inbound request that fans out into several outbound calls, such as the multiple tool calls from one agent turn, can exhaust the downstream budget long before it uses up the inbound one. Treating all of this as a single undifferentiated limit hides exactly the fan-out cost that matters most.

API Rate Limiting Best Practices​

Start conservative and communicate limits clearly​

It's easier to raise a limit later than to walk one back after clients have built workflows around a generous default. Publish the limit in your API documentation and in response headers, so clients can design around it instead of discovering it through trial and error.

Use tiered limits by plan or client type​

One global limit rarely suits every consumer. Internal services, partner integrations, and free-tier public clients tend to have genuinely different usage patterns. Tiered limits, tied to a subscription plan, an API key type, or an authenticated role, let you give higher throughput to consumers who need it without opening the same headroom to everyone.

Monitor and adjust based on real usage​

A limit set once during initial design tends to drift out of step with how the API is actually used. Track 429 rates per client and per endpoint. A client that consistently hits its limit is either a sign the limit needs revisiting, or a client that's polling when it should be using push notifications or webhooks. An endpoint with a disproportionate share of 429s relative to its actual resource cost suggests the limit was set without enough data.

This is also where the algorithm you chose starts to matter in practice, not just in theory. A fixed-window limiter that looks fine in aggregate metrics can still let a client burst to twice its intended rate right at the window boundary. If your monitoring shows request spikes clustered at minute or hour boundaries, that's usually the giveaway, and it's a reason to move that particular limit to a sliding window or token bucket rather than simply raising the number.

Rate Limiting Beyond Traditional APIs: LLM and AI Token Limits​

Everything so far assumes that a request is a request: one unit of consumption regardless of what it contains. That assumption breaks down for LLM and AI traffic, where a single call to a model provider can cost very different amounts of compute depending on prompt length, the model chosen, and how much the model generates in response. A one-line completion and a fifty-page document summary can both count as "one request" against a request-count limit while consuming wildly different resources.

That's why token-based rate limiting exists as a separate control alongside request-count limiting. Instead of capping requests per window, it caps tokens per window, the units a model consumes and generates. This tracks actual cost far more closely than a request count does. The underlying problem is the same one request-count limiting solves, stopping one client from taking a disproportionate share of a shared, finite resource. Only the unit of measurement changes, and the resource being protected is model spend as much as server capacity.

It's worth being precise about the traffic path, because it differs from the inbound client requests described earlier. A user or application sends a request inbound to your API, and your API (or an AI gateway in front of it) makes an outbound call to the LLM provider on that request's behalf.

Token-based limiting isn't restricted to that outbound leg, though. A platform can budget an inbound consumer's token usage, meaning how many tokens a given application or API key may spend against your AI-powered endpoint, separately from protecting the outbound allowance you have with the model provider. Production setups often need both. Within that, it helps to distinguish three kinds of ceiling:

1. A per-request token cap, which rejects or truncates any single call that asks for too much.
2. A tokens-per-interval quota, the same idea as request-count rate limiting but measured in tokens instead of requests.
3. A monetary spend budget, a currency ceiling, since different models price input and output tokens differently.

Which of these a given platform supports, and whether it counts input and output tokens separately, varies by product.

The practical trigger for adopting token-based limits isn't a particular volume threshold, because token cost per request varies too widely for a flat number to mean much across teams. More reliable signals are architectural and organizational: routing across more than one model or provider, needing to attribute spend to specific teams or departments, having regulated or sensitive data flowing through prompts, or simply running at a scale where a handful of misconfigured clients could noticeably move the monthly bill. Any one of those is a better reason to add token-based controls than an arbitrary monthly token count. For a wider look at controlling LLM spend beyond rate limiting, see WSO2's guide to LLM cost control.

How WSO2 Implements API Rate Limiting​

The WSO2 API Platform treats REST, GraphQL, and AI/LLM traffic as related but distinct traffic paths managed from one control plane, rather than forcing every type of traffic through the same policy model.

For traditional API traffic, the platform's API Gateway applies request-count limits to REST, GraphQL, and other synchronous traffic, alongside authentication and request/response policies. The Rate Limiting policy can be configured at two levels:

• API level, where one request-count limit applies to every operation in the API.
• Resource level, where individual operations carry their own tighter limits, which is useful when one endpoint is much more expensive than the rest of the API.

Requests over the limit are rejected with 429 Too Many Requests.

For more control, the Advanced Rate Limit policy adds a choice of GCRA or fixed-window algorithms, and in-memory or Redis-backed counters for limits shared across gateway replicas, with a fail-open or fail-closed setting for when Redis is unavailable. It also returns response headers in both the X-RateLimit-* and IETF RateLimit-* styles, plus Retry-After on 429s, all enabled by default. Tiered limits per consumer come from subscription plans.

For AI and LLM traffic, the platform's AI Gateway applies the token-based model described in the previous section. Its LLM Proxy capability enforces token-based rate limits on traffic to model providers. These limits can be attached to a single LLM proxy (one application's budget) or to an LLM provider (a ceiling shared by every proxy that uses it). Because token counts come from the model's response, the limit takes effect in the response phase: the call that crosses the threshold completes, and the next one is rejected.

For budgets in currency rather than tokens, the LLM Cost and LLM Cost-Based Rate Limit policies price each call and stop traffic once a spend quota is used up. AI spend can then be capped per application or per provider, instead of being treated as one undifferentiated line item.

Because API-level and token-based limits are configured in the same platform rather than in two disconnected tools, a team running both traditional REST APIs and LLM-backed services can apply consistent governance, with request-count limits on one traffic path and token limits on the other, without maintaining a separate system for each. That's a deliberately bounded claim: the platform gives you one place to configure and observe both controls, not a guarantee that every failure mode disappears because the tooling is unified.

If you're deciding where to start, the practical entry points are the WSO2 API Platform policy reference, which covers API-level and resource-level limits, and the token-based rate limiting guide for LLM traffic.

Conclusion​

Rate limiting is one of the few controls that protects backend stability, controls cost, and enforces fairness across clients all at once, which is why it appears on almost every serious API security checklist. The mechanics (fixed versus sliding windows, token bucket versus leaky bucket, request counts versus token counts) matter less than getting the fundamentals right: choose a sensible default, communicate it clearly, and watch how it performs against real traffic before tightening or loosening it.

As more of that traffic becomes AI and LLM calls rather than plain REST requests, the same discipline applies, just measured in tokens instead of requests. If you're setting this up across both traditional APIs and AI services, the WSO2 API Platform is built to manage both from a single control plane rather than as separate, disconnected systems.

Frequently Asked Questions​

What is a good rate limit for an API?​

There's no universal number. A sensible starting point comes from your backend's actual capacity, your current traffic patterns, and how much headroom you want for legitimate bursts. Then adjust it based on 429 rates once real usage data comes in.

How do I fix "API rate limit exceeded"?​

Read the Retry-After header if the response includes one, and wait that long before retrying. If you keep hitting the limit, add exponential backoff with jitter to your retry logic and check whether you're making more requests than you need to. Batching calls or caching responses often reduces request volume more effectively than asking for a higher limit.

Is HTTP 429 a rate limit response?​

Yes. 429 Too Many Requests is the standard HTTP status code for a rate-limited or throttled request, defined in RFC 6585.

How do I rate limit an API to 10 requests per minute?​

Configure a fixed-window or sliding-window limiter with a count of 10 and a window of 60 seconds, tracked per client identifier (API key, token subject, or IP address). Most API gateways expose this as a configuration setting, so you don't have to build it from scratch. In WSO2 API Platform, for example, it's a Rate Limiting policy set to 10 requests per minute, or the Advanced Rate Limit policy's fixed-window or GCRA algorithm if you need more control.