LLM Rate Limiting and 429 Retries in Production: OpenAI, Anthropic, and Gemini (2026)

Updated: July 29, 2026

LLM rate limiting is the mechanism providers use to cap how many requests and tokens your API key can spend per minute. A well-designed production client absorbs the resulting HTTP 429 responses with exponential backoff plus jitter, a client-side token bucket that tracks both requests-per-minute (RPM) and tokens-per-minute (TPM), and a fallback chain that reroutes to a mirror model or a different provider once the local retry budget is spent. I've shipped six of these systems now, and every one of them started with the same mistake: retrying blindly. So, here's what actually holds up in prod for OpenAI, Anthropic, and Gemini in 2026.

• OpenAI, Anthropic, and Gemini all enforce RPM and TPM separately. You hit whichever fires first, and TPM bottlenecks agent workloads because every turn re-sends the full conversation.
• Anthropic's May 2026 compute expansion roughly doubled paid-tier limits, and Claude 4.5 accounts commonly reach 1M+ TPM with review. But rate limits are still per API key, not per account.
• Always honor the Retry-After header first; fall back to exponential backoff with full jitter only when the header is absent. Cap total retry time at your user's request deadline, not at some abstract max.
• Distinguish 429 (your quota) from 529 (provider overload). 529s don't count against your budget the same way, but they still need bounded retry with a hard circuit breaker.
• Failed requests still consume rate-limit slots on OpenAI. A naive tight retry loop makes throttling worse, not better.
• Past ~100 sustained RPS, in-process backoff isn't enough. You need a central queue, per-tenant fair scheduling, and a multi-provider fallback wired through an AI gateway.

Why LLM rate limits melt production apps

The first LLM app I put into production ran fine in staging and started returning 429s to real users about ninety seconds after launch. That's the shape of this failure mode: it doesn't show up until you have concurrency, and by then it's on the incident channel. Rate limits from OpenAI, Anthropic, and Gemini aren't just protection for the provider. They're the load-shedding primitive your own app has to design around, because the moment one team's batch job spikes TPM, every other feature sharing that key starts failing at once.

Rate limits enforce multiple dimensions at once. OpenAI polices RPM, TPM, requests-per-day, and (for some models) tokens-per-day. Anthropic separates input tokens per minute (ITPM) from output tokens per minute (OTPM), which especially punishes agent workloads because every turn ships the full conversation as input. Google's Gemini API layers a similar RPM/TPM model with additional per-model quotas surfaced through Vertex AI. In practice you'll blow the tightest of these limits long before the others are anywhere near their ceiling. A chatbot with short prompts hits RPM. A long-document summarizer hits input TPM. An agent loop that re-ships context every step hits output TPM even when the caller thinks each response is small.

The consequence for your app is boring but important: 429s are a signal, not an exception. Treat them as flow-control feedback from the provider and your architecture stays sane. Treat them as errors to bubble up and users see spinners.

OpenAI, Anthropic, and Gemini rate limits in 2026

Rate limit tiers changed materially in the first half of 2026, and if your runbook still cites 2024 numbers it's misleading. Here's the current shape.

OpenAI runs a five-tier system that auto-promotes based on cumulative spend, enforced at the organization level rather than per key. Tier 1 (>=$5 lifetime spend) gives you roughly 500 RPM and 30,000 TPM on the current GPT-4o class models, which is low by design and will not survive real load. Tier 5 pushes into the tens of thousands of RPM and tens of millions of TPM. You don't need to file a ticket to move up. The promotion is automatic when your usage crosses the threshold, and you can check your current position at platform.openai.com rate limits documentation.

Anthropic assigns limits by account tier (Free, Build, Scale, Custom) and enforces them per API key rather than per organization. The May 2026 compute expansion roughly doubled TPM and RPM ceilings across paid tiers. Claude 4.5 Sonnet and Opus accounts commonly reach 1M+ ITPM under Scale-tier review, and the fast-mode research preview on Opus 5/4.8 exposes a separate rate-limit pool with its own anthropic-fast-* headers. See the Anthropic rate limits reference for the current per-model matrix.

Google Gemini uses a similar RPM/TPM scheme via AI Studio and Vertex AI, with the wrinkle that quota is often visible and adjustable in Google Cloud IAM rather than a per-key dashboard. This makes multi-project deployments powerful but confusing; noisy neighbors within the same project can starve production.

Dimension	OpenAI	Anthropic	Google Gemini
Enforcement scope	Organization	Per API key	Per Google Cloud project
Primary dimensions	RPM, TPM, RPD, TPD	RPM, ITPM, OTPM	RPM, TPM, RPD
Tier progression	Automatic on spend	Manual review for Scale+	Quota increase request
Overload status code	429 (+ 503 rarely)	429 and 529	429 and 503
Retry-After header	Present on 429	Present on 429/529	Present on 429
Remaining-quota headers	x-ratelimit-remaining-*	anthropic-ratelimit-*	Not standard

The takeaway isn't which provider is "best". It's that a production client has to speak all three dialects fluently, because the moment you build a fallback chain you'll be handling their headers in one code path.

Reading the response: 429, 529, Retry-After, and provider headers

Before you write a single line of retry code, teach your client to actually read the response. Every provider gives you enough information to make an informed retry decision, and most production apps throw all of it away.

When OpenAI returns a 429, the body tells you which dimension you exceeded, your current usage, your limit, and how long to wait. A typical message reads "Rate limit reached for gpt-4o on tokens per min. Limit: 30000, Used: 30000, Requested: 1500. Please try again in 3s." That "3s" isn't advisory. It's the reset window for your token bucket. The Retry-After header carries the same signal in machine-readable form. If your client waits 60 seconds when the provider says 3, you're wasting user patience for no reason.

Anthropic layers on a more useful set of headers. On any 429, look for:

• retry-after: seconds to wait before the next attempt
• anthropic-ratelimit-requests-remaining and -reset: how many RPM slots you have left
• anthropic-ratelimit-input-tokens-remaining: remaining ITPM budget
• anthropic-ratelimit-output-tokens-limit, -remaining, -reset: OTPM tracking

The critical distinction on Anthropic is 429 versus 529. A 429 means your quota is exhausted. A 529 means the provider itself is overloaded across all users and your quota has nothing to do with it. Same symptom, completely different remediation. For 429 you back off and eventually route to a fallback; for 529 you can retry sooner but must add a circuit breaker so you don't stack retries against a genuinely down endpoint.

Here's a small helper that normalizes the retry hint across providers. Use it as the first thing your retry decorator consults:

from typing import Optional
import httpx

def parse_retry_hint(resp: httpx.Response) -> Optional[float]:
    """Return seconds to wait before retry, or None if no hint is present."""
    # Standard HTTP header, honored by OpenAI, Anthropic, and Gemini
    ra = resp.headers.get("retry-after")
    if ra:
        try:
            return float(ra)
        except ValueError:
            pass  # HTTP-date form; ignore and fall through

    # Anthropic-specific reset epoch
    reset = resp.headers.get("anthropic-ratelimit-input-tokens-reset")
    if reset:
        import time
        return max(0.0, float(reset) - time.time())

    return None

Exponential backoff with jitter, done right

Exponential backoff is the retry pattern where each attempt waits roughly double the previous delay, and jitter is the small random offset added to that delay so a thundering herd of clients doesn't retry in lockstep. If a thousand agents fail at the same millisecond and each waits exactly one second, all thousand retry at the same millisecond and knock the API over again. Jitter breaks the sync.

AWS formalized three variants (Full Jitter, Equal Jitter, and Decorrelated Jitter) in their exponential backoff and jitter analysis, and Full Jitter is the one you should default to. It picks a uniform-random delay in [0, cap] where cap grows exponentially with the attempt number. It's the simplest to reason about and it minimizes contention.

In Python, tenacity gives you this out of the box. The wait strategy you want is wait_random_exponential (Full Jitter) or wait_exponential_jitter when you want a deterministic base plus a random offset. Skip plain wait_exponential: it has no jitter and it will bite you the first time you have real concurrency. The tenacity API reference covers the full option set.

import httpx
from tenacity import (
    retry, retry_if_exception_type, wait_random_exponential,
    stop_after_attempt, before_sleep_log,
)
import logging

log = logging.getLogger("llm.retry")

class TransientLLMError(Exception):
    """Raised for retriable provider failures (429, 529, 5xx, timeout)."""

def _classify(resp: httpx.Response) -> None:
    if resp.status_code == 429 or resp.status_code == 529:
        raise TransientLLMError(f"throttle {resp.status_code}")
    if 500 <= resp.status_code < 600:
        raise TransientLLMError(f"server {resp.status_code}")
    resp.raise_for_status()

@retry(
    retry=retry_if_exception_type((TransientLLMError, httpx.TimeoutException)),
    # Full Jitter: uniform random in [0, min(cap, base * 2**attempt)]
    wait=wait_random_exponential(multiplier=1, max=30),
    stop=stop_after_attempt(5),
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,
)
def call_llm(client: httpx.Client, payload: dict) -> dict:
    resp = client.post("/v1/messages", json=payload, timeout=60.0)
    # Honor server hint first, then fall through to Tenacity's wait strategy
    hint = parse_retry_hint(resp)
    if hint is not None and resp.status_code in (429, 529):
        import time; time.sleep(hint)
        raise TransientLLMError(f"honored retry-after={hint}s")
    _classify(resp)
    return resp.json()

A few production notes on that snippet. First, reraise=True preserves the underlying exception when the retry budget is spent, which is critical for observability, because you want to see the actual 429 in your traces, not a generic RetryError. Second, always pair the decorator with a hard timeout (that timeout=60.0 on the HTTP call). Retry logic without a per-request deadline turns tail latency into unbounded wait. Third, cap total attempts at 3-5 for user-facing paths; batch workers can go to 7 with a longer overall deadline.

Client-side token bucket for RPM and TPM

Retry-on-429 is reactive. The next level up is proactive: refuse to send a request you already know will be throttled, and hold it until the bucket has capacity. This is the client-side token bucket algorithm, and honestly, it's the single most effective change you can make to an app that's chronically bumping against limits.

The idea is exactly what it sounds like. Maintain a bucket with a maximum capacity and refill it at a steady rate. Each request consumes tokens; if the bucket is empty, the request waits until enough have refilled. The critical mistake almost every implementation makes is tracking only RPM. LLM providers police RPM and TPM independently, so your bucket has to as well. That usually means two buckets, checked together, and both consumed on every call.

import asyncio
import time
from dataclasses import dataclass

@dataclass
class Bucket:
    capacity: float          # max tokens the bucket holds
    refill_per_sec: float    # tokens added per second
    tokens: float = 0.0
    updated_at: float = 0.0

    def __post_init__(self):
        self.tokens = self.capacity
        self.updated_at = time.monotonic()

    def _refill(self) -> None:
        now = time.monotonic()
        elapsed = now - self.updated_at
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_sec)
        self.updated_at = now

    def try_consume(self, n: float) -> float:
        """Consume n tokens; return 0 if allowed, else seconds to wait."""
        self._refill()
        if self.tokens >= n:
            self.tokens -= n
            return 0.0
        deficit = n - self.tokens
        return deficit / self.refill_per_sec

class LLMRateLimiter:
    """Enforces both RPM and TPM before a request leaves the client."""
    def __init__(self, rpm: int, tpm: int):
        self.req_bucket = Bucket(capacity=rpm, refill_per_sec=rpm / 60.0)
        self.tok_bucket = Bucket(capacity=tpm, refill_per_sec=tpm / 60.0)
        self._lock = asyncio.Lock()

    async def acquire(self, est_tokens: int) -> None:
        while True:
            async with self._lock:
                wait_req = self.req_bucket.try_consume(1)
                wait_tok = self.tok_bucket.try_consume(est_tokens)
                if wait_req == 0 and wait_tok == 0:
                    return
                # Roll back partial consumption so we don't double-charge
                if wait_req == 0:
                    self.req_bucket.tokens += 1
                if wait_tok == 0:
                    self.tok_bucket.tokens += est_tokens
                wait = max(wait_req, wait_tok)
            await asyncio.sleep(wait)

limiter = LLMRateLimiter(rpm=500, tpm=30_000)

async def bounded_call(payload: dict, est_tokens: int) -> dict:
    await limiter.acquire(est_tokens)
    return await call_llm_async(payload)

Two subtleties matter. First, est_tokens should be an overestimate. Use your tokenizer on the prompt and add your max_tokens ceiling. This over-provisions the bucket slightly but prevents you from bursting past TPM on responses larger than expected. Second, the rollback in the acquire loop matters: if you consume from the RPM bucket but the TPM bucket blocks, you'd otherwise lose that request slot forever. In my last project this was the bug in our first implementation, and it took nearly a week to spot.

If you're horizontal-scaling this across multiple pods, move the bucket state into Redis with an atomic Lua script. Splitting a shared quota across independent in-memory limiters is the fastest way to double-throttle yourself. For very high volume, offloading eligible traffic to the OpenAI and Anthropic batch APIs takes it out of your synchronous RPM budget entirely.

Should you retry on 500s, timeouts, and 529s?

Yes, but with different rules than 429. This distinction is where I've watched more incidents happen than anywhere else in the retry stack.

A 429 is deterministic: your quota is used up, and you know exactly when it resets. Back off, respect the header, retry. A 500-class response is nondeterministic. The provider had a transient failure, the same request may or may not work next time, and blindly retrying non-idempotent operations can double-charge or trigger duplicate tool calls. Timeouts are the worst of both worlds; you don't know if the request landed and the response was lost, or if it never landed at all.

The rule I use in production:

• 429: retry with the hinted delay, up to your total request deadline. Failed 429 requests still count against OpenAI's quota, so a tight retry loop makes the problem worse. Always sleep the hint, not zero.
• 529 (Anthropic overload): retry with exponential backoff, but wrap in a circuit breaker. If 5 consecutive 529s from the same model, open the circuit for 30 seconds and route to a fallback.
• 500, 502, 503, 504: retry with exponential backoff, max 3 attempts, and only if the request is idempotent. For streaming completions with tool calls, treat non-idempotency as the default and stop after the first retry.
• Read timeouts on streaming: never retry. The completion may have already been generated and billed. Surface the error, don't stack a second bill.
• Connection timeouts: safe to retry, because the request never left the client.

Concurrency, semaphores, and per-tenant fair queueing

The single-node version of concurrency control is an asyncio.Semaphore. Wrap every LLM call in one, size it to your tier's headroom, and you've eliminated roughly 90% of 429s in batch jobs, because you're no longer sending 200 requests to a Tier 1 key with 500 RPM. Start conservative (10 concurrent calls) and raise it only when observability confirms you have room.

import asyncio

_llm_sem = asyncio.Semaphore(10)

async def call_llm_async(payload: dict) -> dict:
    async with _llm_sem:
        return await bounded_call(payload, est_tokens=_estimate(payload))

The multi-tenant version is where it gets interesting. Once you have two internal callers sharing one API key, a batch job from Team A will happily consume every 429-free slot and starve Team B's user-facing traffic. You need admission control that's fair by tenant, not FIFO by request.

The pattern that scales is a central request queue with per-tenant virtual budgets. Each tenant gets an explicit share of the organization's RPM and TPM. Admission control checks tenant budget before enqueuing; a scheduler dequeues based on tenant priority weights; the execution layer calls the provider and applies the token-bucket limiter above. If you're not ready to build this in-house, drop a gateway in front. Options like LiteLLM, OpenRouter, and Portkey all implement virtual keys, per-tenant limits, and fallback rules natively.

Multi-provider fallback and circuit breakers

Somewhere past 100 sustained RPS on a single provider tier, in-process retry stops being enough. The right architecture is multi-provider from day one: a primary, a mirror on a different infrastructure surface (Azure OpenAI is the classic mirror for OpenAI because it's the same model family with a separate rate-limit pool), and a tertiary of last resort (Bedrock, Vertex, or a smaller open-weights model behind vLLM).

Fallback belongs behind a circuit breaker. Without one, a rate-limited primary sees you piling more requests on top of every retry, and you're wasting latency budget on a provider that's already told you no. The breaker's job is to trip after N consecutive throttles, hold open for a cooldown window, then send a single probe request before fully reopening.

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"    # normal traffic
    OPEN = "open"        # skip provider, fail fast
    HALF_OPEN = "half"   # single probe request

class Breaker:
    def __init__(self, failure_threshold: int = 5, cooldown_sec: float = 30.0):
        self.failure_threshold = failure_threshold
        self.cooldown_sec = cooldown_sec
        self.state = CircuitState.CLOSED
        self.failures = 0
        self.opened_at = 0.0

    def allow(self) -> bool:
        if self.state == CircuitState.OPEN:
            if time.monotonic() - self.opened_at >= self.cooldown_sec:
                self.state = CircuitState.HALF_OPEN
                return True
            return False
        return True

    def record_success(self) -> None:
        self.failures = 0
        self.state = CircuitState.CLOSED

    def record_failure(self) -> None:
        self.failures += 1
        if self.failures >= self.failure_threshold or self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN
            self.opened_at = time.monotonic()

PROVIDERS = [
    ("openai", openai_client, Breaker()),
    ("azure", azure_client, Breaker()),
    ("anthropic", anthropic_client, Breaker()),
]

async def call_with_fallback(payload: dict) -> dict:
    last_error = None
    for name, client, breaker in PROVIDERS:
        if not breaker.allow():
            continue
        try:
            result = await client.call(payload)
            breaker.record_success()
            return result
        except TransientLLMError as e:
            breaker.record_failure()
            last_error = e
            continue
    raise RuntimeError(f"all providers throttled or unhealthy: {last_error}")

The mirror pattern only works if your prompts are model-portable. If your prompt depends on a specific model's system-prompt handling or its structured-output quirks, the fallback will produce garbage. Test the fallback path in staging, not during an incident. Anecdotally, prompts that fall back cleanly are also the ones that cache cleanly across providers. The discipline overlaps.

Observability: the dashboard you'll wish you built first

Here is the dashboard you will wish you had built first. Not the one you'll be told to build after your first outage, but the one that would have prevented it. Every LLM app in production needs, at minimum, four charts and one alert.

1. 429 rate by provider, model, and tenant. If this is above 1% for user-facing paths, you're already degrading. Break it out by tenant so you can see when one team is eating the whole budget.
2. p50 and p99 retry-latency overhead. Measure the wall-clock time added by your retry logic separately from actual model latency. When p99 overhead crosses 2 seconds, your retry cap is too aggressive.
3. Token bucket saturation. Emit gauge metrics for each bucket's fill level. When RPM saturation is chronically above 80%, request a tier bump before the outage.
4. Circuit-breaker state transitions. Every open/close is a story; log it with the triggering provider and error class.
5. Alert: fallback usage. If more than 5% of requests are landing on the tertiary provider, page someone. The primary is probably unhealthy in a way retries alone won't fix.

Everything above is table-stakes for a serious LLM ops setup. If you're already instrumenting traces, this fits inside your existing LLM observability stack. Every provider call becomes a span with attributes for provider, model, retry count, and outcome. From there, correlate 429 spikes with tenant IDs and you'll find the noisy neighbor in a minute instead of an hour.

Frequently Asked Questions

How do you fix a 429 error from OpenAI?

Read the Retry-After header (or the "please try again in Ns" hint in the body) and wait exactly that long before your next call. If the error keeps recurring, add exponential backoff with jitter for repeated failures, cap concurrency with an asyncio.Semaphore, and check whether you've been pushed to a lower tier due to a payment method change.

What is the difference between RPM and TPM rate limits?

RPM (requests per minute) counts distinct API calls; TPM (tokens per minute) counts the total tokens across all those calls. They're enforced independently, so a chatbot with short prompts hits RPM first while a long-document summarizer hits TPM first. Anthropic further splits TPM into ITPM (input) and OTPM (output).

Should you retry LLM API calls on 500 errors?

Yes for idempotent requests, up to about three attempts with exponential backoff. For streaming responses where you may have already received partial content, do not retry, because you risk being billed twice for the same completion. Never retry read timeouts on streaming; only connect timeouts are safe.

Do failed LLM requests count against rate limits?

On OpenAI, yes. A request that returns 429 still consumed one request slot and any tokens it carried, which is why blind tight-loop retries make throttling worse. Anthropic and Gemini also count failed requests toward RPM in most cases. Always honor the retry hint and use jittered backoff to avoid amplifying the problem.

What is the best retry library for Python LLM clients?

Tenacity is the default choice. Use wait_random_exponential (Full Jitter) for contention scenarios and wait_exponential_jitter when you want a deterministic base plus randomness. Combine it with a per-request timeout and reraise=True so the original 429 stays visible in your traces.