LLM Rate Limiting: Token Buckets and Concurrency Control

Often, the first HTTP 429 an LLM feature hits comes from the provider, not from anything the team configured. The request count looked healthy. What ran out was the token allowance, often because one workload started sending much larger prompts than everything else sharing the key.

LLM rate limiting caps how much model capacity a client can consume in a time window, measured in tokens as well as requests. This guide covers the limit types that matter and how token buckets behave in practice. It also covers quota scoping per user, team, and feature, and why the gateway is usually the right enforcement point.

Why Standard Rate Limiting Doesn't Work for LLMs​

Traditional rate limiting counts requests. That model works when every request costs the backend roughly the same amount of work. LLM requests don't. One call carries a short question, and the next carries a full document plus a long conversation history. Chat and agent workloads make this worse over time, because each turn usually resends the earlier messages as context. A session that starts small can grow heavier with every exchange while its request rate stays flat.

Providers already meter the difference. OpenAI enforces limits on requests per minute and tokens per minute, among other dimensions, and a client can exhaust whichever one it reaches first. Anthropic goes a step further and meters input tokens and output tokens separately, each with its own per-minute ceiling.

A request-only limit therefore watches the wrong dimension. It lets one heavy client drain a shared token allowance while staying under its request cap. It can also throttle a chatty client whose small calls were never the problem. The symptom is familiar: your own limits report that everything is fine, and the provider starts returning 429s anyway. Token-aware rate limiting closes that gap by counting what the model actually processed.

What You Actually Need to Limit​

Request limits cap how many calls a client makes in a window, which protects against connection floods and retry storms. Token limits cap how many tokens a client consumes in a window, which tracks compute load and cost directly. Most production LLM applications need both, plus a spend ceiling over a longer period.

Cost limits exist as a separate control because tokens don't convert evenly into money. The same token count costs very different amounts on different models, so a token cap alone can't enforce a budget.

Layered windows combine a short window for burst control with a long window for total usage. A per-minute token cap stops a sudden spike, while a daily or monthly cap catches a slow leak that never trips the per-minute limit.

Concurrency deserves its own attention. Request-rate limits control how fast calls arrive, and token limits control how much those calls consume. Neither one caps how many requests are in flight at the same time. A client that sends ten requests per second, each lasting a minute, can hold about 600 open requests under steady load. A per-minute quota alone can't stop that. Use a maximum-in-flight limit for concurrency, with a bounded queue and an admission timeout. Long-lived streams hold a concurrency slot until they complete or the client cancels them. Anthropic notes that a per-minute limit may be enforced over shorter intervals, so a burst can fail even when the per-minute average looks safe. A request limit tuned only to the minute view can miss exactly that burst.

Token Buckets: How Token-Based Limiting Actually Works​

A token bucket holds an allowance that refills at a steady rate up to a fixed capacity. The algorithm works with any unit. In an LLM limiter, the unit is usually the model token, and each request draws down the bucket by the tokens charged to it. That charge can be an estimate made at admission, the usage reported after the response, or an estimate that is reconciled later. When the bucket is empty, further requests receive a 429 until enough capacity refills. Anthropic uses this algorithm for its own API, so capacity replenishes continuously instead of resetting at fixed intervals.

A small, idealized example makes the mechanics concrete. Suppose a bucket holds 60,000 tokens and refills at 1,000 tokens per second. A client that has been idle can spend the full 60,000 in one burst. After that, it can sustain about 1,000 tokens per second, and any faster pace drains the bucket until requests start failing. The capacity sets the burst size, and the refill rate sets the sustained throughput. Real implementations add rules on top of this, such as how input is estimated before the call and when output is counted.

A fixed window resets the whole allowance at a boundary. A client can spend a full allowance just before the reset and another one just after it. A token bucket smooths that out because capacity only returns at the refill rate.

The refill math is the easy part. Three operational details matter more:

• The real count arrives after the response

  Prompt tokens can be estimated before the call, but completion tokens are unknown until the model finishes. A gateway that enforces on actual usage reads the count from the provider's response. WSO2's AI Gateway applies token limits in the response phase. The request that crosses the threshold still reaches the model, and the gateway rejects the next one. Under concurrency, the overshoot can be larger than one request. Every request admitted before the counter updates still runs. If ten requests start together while the window has room for one, all ten complete and are counted afterwards.

  Size limits with that overshoot in mind, especially for workloads with long completions. The alternative is to estimate up front and reconcile later. Anthropic, for example, estimates input tokens at the start of a request and adjusts the count as the request runs. Estimation blocks spend earlier but can reject a request that would have fit. Counting after the response meters what the provider reported, but it isn't perfectly exact. Usage can be missing, arrive late, or differ from how the provider counts against its own limits. Anthropic, for instance, excludes most cached input tokens from its input-token limit. Usage metering suits accounting and budgets, while conservative estimation at admission suits hard ceilings.

• Streaming changes when usage shows up.

  On a streamed response, token usage arrives in the final events rather than in every chunk. On OpenAI Chat Completions and compatible endpoints, the stream omits usage unless the client sets stream_options.include_usage, as WSO2's AI Gateway documentation notes. Other APIs and providers report streamed usage in their own way, so check the one you call. If that usage block never arrives, the limiter has nothing to count.

• Prompt, completion, and total tokens can have separate limits.

  Input and output load behave differently. Retrieval-heavy features inflate prompt tokens, while report generation inflates completion tokens. Gateways that support this, including WSO2's AI Gateway, enforce the most restrictive limit when several are configured.

Per-User, Per-Team, and Per-Feature Quotas​

Provider limits on LLM APIs are scoped to your organization or project, not to your users. OpenAI states this directly in its rate limit documentation. Every user, agent, and feature behind one provider key draws from the same pool. Without tenant-level isolation, one runaway agent loop, misconfigured retry, or heavy batch job can exhaust the shared budget and degrade service for everyone.

The standard production pattern is to limit by identity. Each consuming application gets its own gateway-issued key, which some platforms call a virtual key. Limits attach to that identity rather than to the provider credential. Three scopes cover most cases:

• Per user or API key: Stops a single caller from monopolizing capacity. This matters most for customer-facing features and multi-tenant products.
• Per team: Caps a department's consumption and makes internal chargeback possible.
• Per feature: Separates an interactive assistant from a nightly summarization job, so batch work can't starve the latency-sensitive path.

These scopes usually nest. A per-key limit sits inside a per-team limit, and both sit inside an organization-wide ceiling. Setting that outer ceiling below the provider's own limit reduces upstream throttling, because your gateway rejects most excess traffic first. It doesn't eliminate provider 429s. Burst windows, token estimates, cached-token rules, other consumers of the same account, and shared model pools can all differ from your gateway's view. Keeping most rejections on your side still helps, because that is where you control the headers, the message, and the logging.

The operational payoff is attribution. When one tenant exhausts its quota, the failure stays local and visible. That team sees a 429 on its own key instead of a platform-wide slowdown that nobody can explain. The same per-identity counters also feed spend reporting, which is the foundation of a broader LLM cost control framework.

Application-Level vs. Gateway-Level Implementation​

The useful framing is where enforcement should live, not which approach wins. Application-level limiting gives full control and full business context. Gateway-level limiting gives one consistent policy across every service that shares a provider or a budget. An AI gateway is the shared layer where that policy runs.

Application-level limits break down as systems grow. The logic gets duplicated, the copies drift, and a limiter inside one service can't see what another service is spending on the same provider key.

That doesn't make application logic useless. Product rules such as "free-plan users get fewer generations per day" often belong in the application, because it holds the plan data. Shared capacity, provider credentials, and cost ceilings belong at the gateway.

It also helps to keep the two directions of traffic distinct:

Inbound:   Application --(gateway-issued key)--> AI gateway
           Identity checks and request limits run before any upstream call.

Outbound:  AI gateway --(provider key)--> LLM provider
           Provider RPM and TPM limits apply on this leg.
           Token usage returns in the response and updates token limits.

Guardrails, caching, and routing are separate policies you attach when you need them. None of them is a required hop for rate limiting to work.

What Happens When Limits Are Hit, and How to Design for It​

A 429 means different things depending on who sent it and why. A 429 raised by your gateway's own policy means a configured limit was reached. That limit may belong to one application, to every application using a provider, or to a spending budget. Use the error body and headers to tell them apart, where your gateway exposes that detail. Retrying immediately won't help, so the client should wait for the window to reset or degrade gracefully. A 429 that originates at the provider usually means throttling on shared capacity, which calls for backoff and possibly for routing the request somewhere else. It isn't always transient. Anthropic, for example, returns a 429 without a retry-after header when an organization reaches its spend cap. Classify the error before retrying, and never route a deliberate budget rejection to another paid provider.

RFC 6585 defines 429 and allows the response to include a Retry-After header. Gateways and providers often add their own rate limit headers as well. Names and formats vary, so parse the ones your gateway actually returns. The client below targets a gateway that reports the window reset as an epoch timestamp in X-RateLimit-Reset, as WSO2's AI Gateway does. Treat it as epoch time only because that is this gateway's documented contract. Provider-specific reset headers use other formats.

import requests
import email.utils
import random
import time


class RetryDeadlineExceeded(Exception):
    """The next permitted attempt falls outside the caller's deadline."""

    def __init__(self, response, wait):
        super().__init__(f"next attempt allowed in {wait:.1f}s, past the deadline")
        self.response = response
        self.wait = wait


def parse_retry_after(value):
    """Return seconds to wait from Retry-After (delay-seconds or HTTP-date)."""
    if not value:
        return None
    value = value.strip()
    if value.isdigit():
        return float(value)
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:
        return None
    return max(0.0, when.timestamp() - time.time())


def is_retryable(resp):
    """Retry transient throttling only. Budget or spend exhaustion is final.

    Adapt this check to the error schema your gateway and providers document.
    """
    try:
        detail = str(resp.json()).lower()
    except ValueError:
        return True
    return not any(w in detail for w in ("budget", "spend", "billing", "credit"))


def call_llm(url, headers, payload, max_attempts=4, deadline_s=30.0):
    """POST to an LLM endpoint and back off on retryable HTTP 429s.

    Returns the first non-429 response, a non-retryable 429, or the last 429
    once attempts run out. Raises RetryDeadlineExceeded when the required wait
    does not fit in the remaining deadline, so the caller can fail fast or
    queue the work. Network errors propagate as requests exceptions.
    """
    deadline = time.monotonic() + deadline_s
    resp = None
    for attempt in range(1, max_attempts + 1):
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise RetryDeadlineExceeded(resp, 0.0)
        # requests timeouts apply per connect/read phase, so this is approximate.
        resp = requests.post(url, headers=headers, json=payload, timeout=remaining)
        if resp.status_code != 429 or not is_retryable(resp):
            return resp
        if attempt == max_attempts:
            return resp  # No attempts left, so do not sleep.
        wait = parse_retry_after(resp.headers.get("Retry-After"))
        if wait is None:
            reset_at = resp.headers.get("X-RateLimit-Reset", "")
            if reset_at.isdigit():  # Epoch seconds, per this gateway's contract.
                wait = max(0.0, int(reset_at) - time.time())
            else:
                wait = min(2 ** (attempt - 1), 8) + random.uniform(0, 1)
        if wait > deadline - time.monotonic():
            raise RetryDeadlineExceeded(resp, wait)
        time.sleep(wait)
    return resp

The random jitter in the fallback branch matters when many clients hit a limit at the same moment. Without it, they all retry on the same schedule and collide again.

The deadline_s guard matters for interactive paths. It bounds the whole operation, not just one sleep, and the function never sleeps after its last attempt. If a tenant's quota resets in several minutes, the function raises instead of sleeping inside a user request, so the caller can return a clear message. Batch jobs can catch the same error, push the work onto a queue and drain it at a pace the quota allows. Official provider SDKs often retry 429s on their own. Stacking this loop on top of SDK retries multiplies the attempts, so keep retries in one layer.

Provider-side 429s are where fallback routing when provider quotas are exhausted earns its place. A gateway can move traffic from a throttled model to another model or provider. In WSO2's AI Gateway, the round robin load balancing policies suspend a model that returns a 5xx or 429. The suspension lasts a configurable period, and later requests go to the rest of the pool. The request that received the error isn't retried by that policy, so the client still needs its own retry handling.

One design rule keeps fallback honest. It should route around provider capacity problems, not around your own tenant quotas. If a team has spent its budget, sending its traffic to a different model defeats the point of the budget.

Observability: The Metrics That Make Rate Limiting Actionable​

Limits you can't observe turn into guesswork. A small set of metrics covers most tuning decisions:

• 429 count, split by source. Gateway-issued and provider-issued 429s need different fixes, so report them separately.
• Token consumption per tenant. Track prompt and completion tokens separately, since each points to a different cause.
• Quota utilization. Consumption as a share of each limit per window. A tenant that sits near its ceiling every day needs either a larger quota or a design review.
• Requests without usage data. Streamed calls that never reported token counts are invisible to token limits.
• Cost per tenant and per model. This is the number finance will ask about.

These metrics also help with debugging. A sudden jump in token spend from one tenant is often a misconfigured agent loop or retry bug rather than organic growth. A useful first check is to compare request volume with the number of distinct users. A loop shows many near-identical requests from very few callers, while real growth shows more people using the feature. Alert on changes rather than fixed thresholds. A tenant whose daily token use doubles overnight deserves a look even if it is still under quota. For dashboards and alerting patterns, see this guide on monitoring token consumption and rate-limit metrics.

Implementing This at the Gateway​

WSO2's AI Gateway is part of the open source WSO2 API Platform. The platform governs API traffic and AI traffic under a single control plane. The AI Gateway mediates outbound LLM traffic, and its request-based limit policies work on any API, not only on LLM traffic. The same request limits can therefore cover both kinds of traffic.

The architecture follows the pattern argued for above. Applications authenticate to the gateway with their own gateway-issued API keys, the virtual keys described earlier. The gateway holds the provider key, tracks token consumption from each response, and blocks new requests once the quota for the window is used up.

The main rate limit policies are:

• Token-Based Rate Limit: Attach it to an LLM proxy to cap one application, or to an LLM provider to cap every proxy that uses it. It supports prompt, completion, and total token limits, and each limit takes a count and a duration such as 1m, 1h, or 24h.
• Rate Limit - Basic and Rate Limit - Advanced: These cap request counts before the call goes upstream. Both support GCRA or fixed-window algorithms and a Redis backend. The advanced policy adds multi-dimensional quotas and weighted limiting.
• LLM Cost-Based Rate Limit: This enforces a spend budget in USD per time window, using cost data from the LLM Cost policy (or the Azure LLM Cost policy for Azure providers).
• Rate limit response headers: X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset let any client read its remaining budget.

Per-team budgets follow from the proxy scope. Give each team or application its own proxy with its own token limit. To attribute usage in analytics, map each proxy's API keys to a GenAI application so dashboards group token usage and cost by that name.

When you size a quota, start from the budget each application actually has. The WSO2 walkthrough uses a deliberately tiny total token limit so that a 429 is easy to trigger. It recommends setting production values from real per-application budgets. A companion sample in the open source wso2/api-platform repository runs the same setup locally against a mock backend, which is a quick way to see the 429 behavior before touching production keys. The practical triggers for centralizing limits are qualitative. Watch for multiple models or providers, a need for spend attribution by team, or regulated data in prompts. Several services sharing one provider account is another strong signal.

Conclusion​

Request limits and token limits solve different problems, and LLM traffic needs both. Request limits cap how many calls start in a short interval. Token limits track what the model actually processed, which is what drives cost and provider throttling. Spend ceilings then cover the gap that token counts leave across models with different prices.

Where those limits run matters as much as how they are sized. Provider quotas are shared across your whole organization, so isolation between users, teams, and features has to come from your side. A gateway gives you one place to issue keys, enforce quotas, read rate limit headers, and see which tenant hit which ceiling. Keep product rules that depend on plan data in the application, and move shared capacity and cost controls to the gateway.

To see the pattern working, follow a step-by-step guide to configuring token-based rate limits on an LLM proxy. Then explore the WSO2 AI Gateway for routing and cost controls on the same traffic.

Frequently Asked Questions​

How do token limits and request limits differ?​

A request limit counts calls per time window. A token limit counts the tokens those calls consume, which maps to cost and compute. A few large prompts can exhaust a token limit while the request count barely moves.

What happens when LLM rate limits are exceeded?​

The API returns HTTP 429 Too Many Requests. Requests stay blocked until the window resets or the bucket refills. Response headers usually indicate when capacity returns.

How do you prevent HTTP 429 errors in production LLM applications?​

You can reduce them but not eliminate them. Keep tenant quotas below provider ceilings, smooth bursts with queues, back off using the reset headers, and route to another model when a provider throttles.

Should rate limiting happen at the application level or the gateway level?​

Use the gateway for shared capacity, provider credentials, and cost ceilings. Keep product rules that depend on plan or user data in the application. Most production systems use both.

Does rate limiting add noticeable latency?​

A local counter check is small next to the time a model takes to respond. A shared counter store such as Redis adds a network round trip, so measure the overhead under your own load.