An LLM circuit breaker is a runtime state machine that stops sending requests to an upstream model provider once its failure rate or latency crosses a threshold, then probes with a small trickle of requests to decide when to close again. In production, you pair it with a fallback chain, an ordered list of providers or models that catches the traffic while the primary is tripped. I've run this pattern across roughly 40M LLM calls a month, and honestly, the two facts that matter are these: naked retries make outages worse, and the incident where you learn this always happens on a Friday.
Circuit breakers protect your app from cascading failures when a provider degrades. Retries alone amplify the incident because every request pays full latency before failing.
Fallback chains route to a secondary provider (Anthropic when OpenAI is down, or a self-hosted Llama for the emergency floor) using a compatibility shim to normalize request and response shapes.
Health scoring beats binary up/down. Score by rolling error rate, p95 latency, and 429 pressure so the breaker trips on partial degradation, not just full outages.
Request hedging (fire the second provider after a delay, cancel the loser) shaves p99 latency by 30–50% but doubles cost on the tail. Budget it.
Streaming failover is the hardest case. You cannot resume mid-stream, so define a "commit point" where partial output is either flushed to the client or discarded.
The dashboard you actually need: per-provider error rate, breaker state timeline, fallback ratio, retry budget consumption, and hedge win rate.
What is an LLM circuit breaker?
A circuit breaker is a pattern borrowed from electrical engineering by way of Netflix Hystrix, and it applies almost cleanly to LLM traffic. The idea: wrap every call to a provider in a state machine that tracks recent outcomes. When failure rate crosses a threshold, the breaker "trips" and stops making calls entirely, either failing fast or routing to a fallback. After a cooldown, it enters a "half-open" state where a limited number of probe requests are allowed. If those succeed, the breaker closes and normal traffic resumes. If they fail, it re-opens.
For LLM apps the analog is direct, but there are three nuances the generic pattern doesn't cover. First, LLM failures are rarely binary. You're often looking at a provider that answers, just slowly, or that starts returning tool-call schemas with wrong argument types after a silent model update. Second, the "failure" surface includes rate limits (HTTP 429s), which are not really failures but back-pressure signals; you want to trip the breaker for the affected model tier but keep hammering another. Third, most LLM SDKs have their own retry logic baked in, so if you're not careful you end up with three layers of retries that make a five-second incident into a five-minute one. This is the mistake I see most often, and it's the one that makes engineers say "we had retries and it still went down."
Why retries alone don't save you during an outage
Imagine OpenAI returns 503s for 90 seconds, a boring, real event that happens roughly monthly. Your app has "resilient" code: three retries with exponential backoff. What actually happens is a retry storm. Every user request that would have made one call now makes four. Your outbound QPS quadruples in the exact window when the provider is asking you to back off. When the provider recovers, your worker pool is still saturated with retries from 30 seconds ago, so genuine new requests queue behind zombie traffic. You see recovery on OpenAI's status page and full latency on yours.
The specific damage: worker starvation, database connection exhaustion (each in-flight request holds a session for whatever record it locked before calling the LLM), and cascading timeouts on upstream services that were waiting on your LLM-backed endpoint. According to the OpenAI status history, elevated error rates for one hour or more happen several times a year, and shorter degradations happen weekly. Assume it. Design for it.
A circuit breaker fixes this by refusing to make calls when the provider is unhealthy. Instead of your requests dying slowly at the provider's edge, they die immediately at your edge and can be routed to a fallback or returned with a graceful degradation. Latency stays bounded. Your worker pool stays free. That's the whole point.
The three states of a circuit breaker
Every LLM circuit breaker moves between three states. Getting the transitions right is where most implementations go wrong.
Closed (normal operation)
Requests flow through to the provider. The breaker keeps a rolling window of outcomes, and I recommend a 60-second sliding window of at least 20 samples. If the failure ratio in that window exceeds a threshold (I use 50% for a hard trip, or a slower "degraded" trip at 25% for 30 consecutive seconds), the breaker transitions to open.
Open (blocked)
All requests to this provider fail immediately with a well-typed exception. Your fallback logic catches that exception and routes elsewhere. The breaker stays open for a cooldown period, typically 15–30 seconds. Don't use a fixed cooldown for long outages. Jitter it and back it off if the half-open probe fails.
Half-open (probing)
After cooldown, the breaker allows a small number of probe requests through, usually one or two concurrent, not the full traffic. If those succeed, the breaker closes. If they fail, back to open with a longer cooldown. The half-open state is what prevents the classic "the provider comes back and we instantly overwhelm it with pent-up demand" thundering herd.
Here's a minimal implementation using pybreaker, which handles the state machine and rolling stats for you.
import pybreaker
from openai import OpenAI, APIError, RateLimitError
# One breaker per provider. Never share.
openai_breaker = pybreaker.CircuitBreaker(
fail_max=10, # 10 failures in the window trips it
reset_timeout=20, # 20s cooldown before half-open
exclude=[RateLimitError], # 429s use a separate throttle, not the breaker
name="openai-gpt-4o",
)
client = OpenAI(max_retries=0, timeout=15.0)
@openai_breaker
def call_openai(messages, model="gpt-4o"):
return client.chat.completions.create(
model=model,
messages=messages,
timeout=10.0,
)
# In your request handler:
try:
resp = call_openai(messages)
except pybreaker.CircuitBreakerError:
# Breaker is open. Go to fallback immediately, don't wait.
resp = call_fallback(messages)
except APIError:
# Provider returned an error but breaker is still closed.
# Counted toward the failure window; also try fallback.
resp = call_fallback(messages)
Notice what's not here: no time.sleep(retry_delay) loop, no manual state tracking. The breaker owns that, and it does so with atomic counters that work under concurrency.
Health scoring: what to actually measure
Binary success/failure isn't enough for LLM providers. You need a health score because providers degrade before they die. The classic signature: p95 latency creeps up over 90 seconds, a small fraction of requests start returning malformed JSON, and only then does the 503 wave arrive. If your breaker only trips on HTTP errors, you've been shipping broken responses to users for the full lead time.
I score providers on four signals, weighted:
Error rate (weight 0.4): 5xx, connection timeouts, and application-level "unusable response" errors like malformed JSON when you asked for structured output.
Latency degradation (weight 0.3): current p95 divided by 30-minute baseline p95. Above 2.0, start deducting score.
429 pressure (weight 0.2): rate of 429s relative to your normal budget. Separate from error rate because it's a back-pressure signal, not a health signal.
Semantic quality (weight 0.1): if you have a lightweight validator on outputs (schema check, refusal detector, tool-call sanity), a rising invalid rate is often the earliest warning.
The combined score determines routing weight, not just a boolean trip. A provider at 0.9 gets normal traffic. At 0.6, it gets 30% of traffic while your fallback picks up the rest. At 0.3, it's tripped. This is called weighted fallback and it beats hard trips for anything with variable load. Related pattern: our take on LLM rate limiting and 429 retries covers the 429-pressure signal in more depth if you're just starting from scratch.
Building a fallback chain across OpenAI, Anthropic, and Gemini
A fallback chain is an ordered list of providers, each behind its own breaker. On failure, you walk the chain until one succeeds. The tricky part isn't the chain itself. It's making three different SDKs return compatible shapes so your downstream code doesn't have to care who answered.
from dataclasses import dataclass
from typing import Callable
import pybreaker
from openai import OpenAI
from anthropic import Anthropic
import google.generativeai as genai
@dataclass
class LlmResponse:
text: str
provider: str
model: str
input_tokens: int
output_tokens: int
def call_openai_impl(messages, model="gpt-4o"):
resp = openai_client.chat.completions.create(
model=model, messages=messages, timeout=10.0
)
return LlmResponse(
text=resp.choices[0].message.content,
provider="openai", model=model,
input_tokens=resp.usage.prompt_tokens,
output_tokens=resp.usage.completion_tokens,
)
def call_anthropic_impl(messages, model="claude-sonnet-5"):
# Anthropic wants system message separately.
system = next((m["content"] for m in messages if m["role"] == "system"), "")
turns = [m for m in messages if m["role"] != "system"]
resp = anthropic_client.messages.create(
model=model, max_tokens=2048, system=system, messages=turns, timeout=10.0,
)
return LlmResponse(
text=resp.content[0].text,
provider="anthropic", model=model,
input_tokens=resp.usage.input_tokens,
output_tokens=resp.usage.output_tokens,
)
def call_gemini_impl(messages, model="gemini-2.5-pro"):
m = genai.GenerativeModel(model)
# Gemini uses "user"/"model" role names.
contents = [
{"role": "user" if x["role"] == "user" else "model",
"parts": [x["content"]]}
for x in messages if x["role"] != "system"
]
resp = m.generate_content(contents, request_options={"timeout": 10.0})
return LlmResponse(
text=resp.text,
provider="google", model=model,
input_tokens=resp.usage_metadata.prompt_token_count,
output_tokens=resp.usage_metadata.candidates_token_count,
)
# One breaker per provider.
BREAKERS = {
"openai": pybreaker.CircuitBreaker(fail_max=10, reset_timeout=20, name="openai"),
"anthropic": pybreaker.CircuitBreaker(fail_max=10, reset_timeout=20, name="anthropic"),
"google": pybreaker.CircuitBreaker(fail_max=10, reset_timeout=20, name="google"),
}
CHAIN = [
("openai", call_openai_impl),
("anthropic", call_anthropic_impl),
("google", call_gemini_impl),
]
def call_with_fallback(messages) -> LlmResponse:
last_exc = None
for name, impl in CHAIN:
breaker = BREAKERS[name]
try:
return breaker.call(impl, messages)
except pybreaker.CircuitBreakerError as e:
last_exc = e
continue # breaker open, try next
except Exception as e:
last_exc = e
continue # provider failed, try next
raise RuntimeError("All providers exhausted") from last_exc
Two things to notice. First, the response normalization. LlmResponse hides SDK differences so consumers don't branch on provider. Second, ordering matters: put your cheapest, best-behaved provider first, not the "smartest." I've watched teams put Claude Opus first "because it's best" and then panic when their bill triples during a routine OpenAI incident that lasted 20 minutes. If you want a managed layer that does this for you, our comparison of LiteLLM, OpenRouter, and Portkey walks through the tradeoffs.
Also, tool calling and structured outputs are not portable across providers. If your primary request uses OpenAI's tool_choice or Anthropic's tool schema, you must transform the request shape when you fall back. Either write a translation layer or accept that fallback is text-only (which is fine, often the fallback is "return a graceful degraded response" rather than "run the full agent workflow"). For deeper patterns here, see our writeup on LLM tool schema design.
Request hedging for tail latency
Circuit breakers protect you when a provider is failing. Request hedging protects you when a provider is slow. The pattern comes from Google's "The Tail at Scale" paper: fire a request to your primary provider, wait N milliseconds, and if you don't have a response yet, fire a second request to a different provider. Return whichever completes first, cancel the loser.
For LLM workloads this can cut p99 latency by 30–50%, because provider tail latency is spiky in a way that model choice barely affects. A slow request is usually a slow shard, not a hard prompt. The cost: on the hedge fraction of requests (typically 5–10% if you set the hedge delay near your p95), you pay for both calls. Budget accordingly.
import asyncio
async def hedged_call(messages, hedge_delay_ms=800):
primary = asyncio.create_task(call_openai_async(messages))
try:
# Wait up to hedge_delay for primary
return await asyncio.wait_for(
asyncio.shield(primary), timeout=hedge_delay_ms / 1000
)
except asyncio.TimeoutError:
# Primary is slow, so hedge with secondary
secondary = asyncio.create_task(call_anthropic_async(messages))
done, pending = await asyncio.wait(
{primary, secondary}, return_when=asyncio.FIRST_COMPLETED,
)
# Cancel the loser. Important for cost and provider load.
for task in pending:
task.cancel()
winner = done.pop()
return winner.result()
Two rules I've made mistakes with and won't again. Set the hedge delay to your p95, not your median, or you'll hedge 50% of requests and double your bill. And cancel the losing request. Most SDKs support this cleanly with task.cancel() plus proper timeout handling, but the provider still charged you for the tokens generated before cancellation, so make sure you're logging that cost.
The bulkhead pattern: isolating provider pools
The bulkhead pattern isolates concurrency per provider so a slow one can't starve your worker pool. Imagine you have 50 async workers and Anthropic starts responding at 30 seconds instead of 2. Without bulkheads, within a minute all 50 workers are blocked on Anthropic and OpenAI traffic queues behind them even though OpenAI is fine. With bulkheads, you allocate (say) 25 workers to Anthropic and 25 to OpenAI. When Anthropic slows, only its 25 saturate; OpenAI keeps flowing.
class ProviderBulkhead:
def __init__(self, max_concurrent):
self.sem = asyncio.Semaphore(max_concurrent)
self.rejected = 0
async def call(self, impl, *args, **kwargs):
# non-blocking acquire. If pool is full, fail fast.
if not self.sem.locked() or self.sem._value > 0:
async with self.sem:
return await impl(*args, **kwargs)
self.rejected += 1
raise BulkheadFullError()
BULKHEADS = {
"openai": ProviderBulkhead(max_concurrent=25),
"anthropic": ProviderBulkhead(max_concurrent=25),
"google": ProviderBulkhead(max_concurrent=10), # smaller quota tier
}
Size each bulkhead based on your provider's rate limit divided by expected request latency. If your TPM budget on OpenAI allows 30 concurrent in-flight requests, set the bulkhead at 25 (leave headroom) and let the 26th caller fall through to Anthropic instead of queuing. Combine bulkhead exhaustion with the circuit breaker: a full bulkhead should degrade the health score, so if it stays full for a minute the breaker trips and traffic flows entirely to the fallback.
Streaming failover: the hardest failure mode
Failover during a streaming response is the case that keeps ops people up at night. You've already sent 40 tokens to the client via server-sent events. The provider connection drops mid-stream. Now what? You can't resume from the same position on a different provider. You'd have to replay the prompt, and the new provider would generate different text. Concatenating "openai-generated intro" with "anthropic-generated middle" produces incoherent output.
There are three viable strategies:
Fail forward with a "commit point": once you've streamed N tokens, you commit and no failover is possible. Any provider failure after commit is either shown to the user as a truncated response with a "regenerate" button, or the buffered output is flushed and you close the stream.
Buffer and switch: hold the first N tokens server-side. If the provider fails within the buffer window, restart against the fallback provider without the client seeing anything. Once the buffer is exhausted (typically 300–500ms), you commit and switch to the fail-forward strategy.
Regenerate on failure: on any streaming failure, terminate the stream, silently retry the whole request against the fallback, and send the second stream in full. Simplest to implement, worst UX for latency.
I default to strategy 2 (buffer and switch) for user-facing chat and strategy 3 (regenerate) for machine-to-machine API workflows where latency matters less than correctness. The buffer approach requires you to accept small latency overhead (you're not streaming the first 300ms even though tokens are ready), but it eliminates the ugly "half a response" case. Our deeper writeup on SSE, backpressure, and cancellation patterns covers the underlying stream mechanics you'll need to build any of these.
The dashboard you'll wish you had built first
Every resilience pattern above generates data. The only way to know whether your fallback chain is doing its job is to graph it. Here are the six panels I put on the top row of the LLM ops dashboard, and why each one has saved me during an incident.
Per-provider error rate (5xx + timeout), split by model. This is the earliest visible signal that a provider is degrading, before the status page updates, before your users complain.
Breaker state timeline, one row per breaker, showing closed/half-open/open transitions. When someone asks "was OpenAI down at 3pm?" you point at this instead of arguing.
Fallback ratio: percentage of requests served by the primary vs each fallback. A slow creep upward means your primary is degrading and you should investigate before it trips.
Retry budget consumption: retries as a fraction of successful requests. Above 10% means you're paying for retries that mostly aren't helping. Tune your backoff.
Hedge win rate: percentage of hedged requests where the secondary beat the primary. If it's under 30%, your hedge delay is too aggressive.
Cost per successful response, per provider. This one catches configuration bugs. A stuck fallback to Claude Opus can quadruple your bill in a day.
You can build these with any observability stack. I use Prometheus for counters and Grafana for the timeline, and instrument the breakers with OpenTelemetry GenAI semantic conventions so the provider, model, and error type are queryable dimensions. If you're just getting started with instrumentation, our guide on LLM observability in production covers the tracing and monitoring foundation you'll build these panels on top of.
None of these patterns are exotic. They're the standard resilience toolkit (circuit breaker, bulkhead, hedging, fallback chain) applied to a specific failure surface that happens to be very expensive and very public when it breaks. The reason to build them now is that LLM providers will keep having incidents. The only question is whether your product notices.
Frequently Asked Questions
What is the difference between a retry and a circuit breaker?
A retry re-sends a failed request, hoping the next attempt succeeds. A circuit breaker refuses to send requests once it observes too many failures, so calls fail immediately at your edge instead of paying the full provider timeout. Use both: retries handle transient blips within a single request; circuit breakers protect your worker pool when the provider has a sustained problem.
How do you handle an OpenAI outage in production?
Wrap OpenAI calls in a per-provider circuit breaker with a 15–30 second cooldown. When the breaker trips, route to a secondary provider (Anthropic or Gemini) via a fallback chain, using a response normalization layer so downstream code doesn't need to know who answered. Don't rely on the OpenAI SDK's built-in retries alone. Turn them off (max_retries=0) and let your resilience layer own the retry policy.
What is LLM request hedging and when should I use it?
Request hedging fires a duplicate request to a second provider after a delay (usually your p95 latency), then returns whichever finishes first and cancels the loser. It's most useful for latency-sensitive user-facing paths where you can afford to double the cost on the tail 5–10% of requests. Skip it for batch or background workloads where extra latency is fine and cost matters more.
Can you failover between providers mid-stream?
Not cleanly. Once you've sent tokens to the client, switching providers would produce incoherent output because the new provider generates different text. The workable pattern is to buffer the first 300–500ms of the stream server-side. If the primary fails inside that window, restart against the fallback; after the buffer commits, any failure terminates the stream and you either show a truncated response or trigger a full regeneration.
How many providers should be in a fallback chain?
Two is the minimum, three is the sweet spot for most production apps: primary, secondary, and an "emergency floor" like a self-hosted Llama or a cheaper model that guarantees an answer. More than three chains rarely pay off. By the time you're on the third fallback, the incident is bad enough that a graceful degraded response is better than another attempt.
Should the circuit breaker trip on 429 rate-limit errors?
No. 429s are back-pressure signals, not failures. The provider is telling you to slow down, not that it's broken. Treat them separately: exclude them from the breaker's failure count, and instead use a token bucket or adaptive concurrency limiter to reshape your request rate. If 429s persist despite throttling, you may want a secondary breaker specifically for rate-limit pressure that routes to the fallback.
Design JSON Schemas for LLM function calling that hold up under real production traffic. Strict mode, enums, nesting depth, Pydantic-first, and the failure modes I measure across OpenAI, Anthropic, and Gemini.
LLM-as-a-Judge in production: cancel positional, verbosity, and self-preference bias, calibrate against Cohen's kappa with 200+ human labels, and wire reliable automated evaluation into your CI pipeline without breaking the budget.
Production LLM streaming over SSE: fix nginx buffering, propagate cancellation to the provider, handle backpressure, and instrument TTFT for OpenAI, Anthropic, and Gemini.
