OpenAI Batch API vs Anthropic Message Batches: A Production Guide to 50% Cheaper LLM Inference (2026)
OpenAI Batch API and Anthropic Message Batches both cut LLM token costs by 50% in exchange for a 24-hour SLA. Here is when batch wins, working Python for both, and the production pitfalls nobody mentions.
The OpenAI Batch API and Anthropic Message Batches API let you submit up to 50,000 (OpenAI) or 100,000 (Anthropic) LLM requests in a single asynchronous job and pay 50% less per token, in exchange for a results SLA of up to 24 hours. In practice, most batches finish in 1–6 hours. So if you run document enrichment, evaluation harnesses, nightly summarization, or any non-interactive workload above roughly $500/month in spend, batch-first is the cheapest production pattern available in 2026, with zero hit to model quality.
Both OpenAI Batch API and Anthropic Message Batches give a flat 50% discount on input and output tokens versus the synchronous endpoints, with identical model quality.
OpenAI accepts up to 50,000 requests or 200 MB per JSONL job; Anthropic accepts up to 100,000 messages per batch and supports a 300K-output-token beta header for long-form generation.
Neither API supports streaming responses or multi-turn tool-use loops inside a batch. Batches are single-shot completions only.
Neither API has webhook completion callbacks at GA. You poll, ideally with exponential backoff (10s, 20s, 40s, cap at 120s).
Batch discounts stack with prompt caching: a long system prompt cached at 90% off, sent through batch at 50% off, can land at ~5% of the headline price on Sonnet 4.6.
Track custom_id in your input JSONL. Output line order is not guaranteed, and you'll need it to join results back to your source records.
When the batch API actually pays off
Batch APIs are the right tool for one specific shape of workload: high-volume, non-interactive, deadline-tolerant inference. If a human is waiting on the response, batch is wrong. If a downstream cron, ETL, or evaluation harness is waiting, batch is almost always right.
From running production LLM pipelines for a few years now, the workloads that benefit most are pretty consistent: nightly document enrichment (classification, extraction, summarization across new records), full-catalog content refreshes, offline evaluation runs against thousands of golden examples, dataset labeling for fine-tuning, embedding precomputation for RAG indexes, and retrospective re-scoring when you change a prompt and need to backfill historical answers.
The workloads where batch is the wrong tool are equally clear. Chatbots, real-time translation widgets, copilots, on-demand agents, anything triggered by a user click. The 24-hour SLA is a hard ceiling, not a soft guideline. If your product breaks when results take six hours, don't use batch. Use synchronous calls with prompt caching and a model router instead.
The break-even math is predictable, too. If a workflow spends more than ~$500/month on synchronous tokens, the 4–8 hours of engineering to refactor it into batch typically pays back inside two months. Below that threshold, the engineering time is more expensive than the savings, and you should just keep the synchronous path.
OpenAI Batch API vs Anthropic Message Batches: feature comparison
At a feature level the two APIs are 90% the same shape (async submission, 50% discount, 24-hour SLA, no streaming, polling-only completion), but they differ in submission ergonomics, batch size, and a few sharp edges that matter in production.
Dimension
OpenAI Batch API
Anthropic Message Batches
Discount
50% on input and output tokens
50% on input, output, cached reads, and cache writes
SLA
24h (most finish 1–6h)
24h (most finish well under)
Max requests per batch
50,000 requests or 200 MB
100,000 requests
Submission format
Upload JSONL to Files API, then create batch
POST requests array directly to batches endpoint
Results retrieval
Download single JSONL output file
Server-Sent Events stream of results
Streaming inside a request
Not supported
Not supported
Multi-turn tool loops
Not supported (single-shot only)
Not supported (single-shot only)
Max output tokens
Model-default (e.g. 16K on GPT-4.1)
Up to 300K with output-300k-2026-03-24 beta header
The biggest practical difference is the submission shape. OpenAI treats batches as a job system on top of the Files API: you write JSONL to disk, upload it, then create a batch that references the file ID. Anthropic treats batches as an API-native object, so you POST an array of fully-formed message requests in a single call. For pipelines that already serialize to JSONL, OpenAI's flow is natural. For pipelines that build requests in memory and submit them directly, Anthropic's is cleaner.
The second difference that bites in production is the 300K output token ceiling on Anthropic batches. If you're generating long-form reports, full-document rewrites, or extended chain-of-thought rationales, the synchronous Sonnet 4.6 and Opus 4.7 endpoints cap at 64K or 128K output tokens, while batch lifts that to 300K. For any "rewrite this entire 80-page document" workload, batch is the only practical path.
How long does the OpenAI Batch API take to run?
Every OpenAI batch job must declare a completion_window of "24h"; any other value gets rejected at submission. That's the SLA ceiling. The observed reality is much faster. Most small-to-medium batches (up to ~10K requests) finish in 1–2 hours, and even 50K-request batches typically complete in 3–6 hours during off-peak windows. End-of-quarter and major-launch days are slower. I've watched batches sit in in_progress for the full 23 hours when global demand spikes, especially the day after a major model launch.
When a batch expires, OpenAI cancels remaining work and returns whatever completed successfully through the output file. You're billed only for completed requests. That makes partial completion a real production case to plan for. Never assume a single batch will return 100% of submitted requests, and design the join step to handle missing custom_ids gracefully.
Anthropic's Message Batches behave similarly. The 24-hour window is the SLA, and most batches complete in minutes to hours depending on model and size. Anthropic doesn't publish guaranteed minimums either, so the same partial-completion planning applies.
Submitting an OpenAI batch job in Python
Here's the full submit-poll-retrieve cycle with the openai Python SDK v1.x. The pattern: build a JSONL file in memory, upload it with purpose="batch", create the batch, poll with exponential backoff, then download the output file and rejoin on custom_id.
import io
import json
import time
from openai import OpenAI
client = OpenAI()
documents = [
{"id": "doc-001", "text": "This product exceeded my expectations."},
{"id": "doc-002", "text": "Shipping was slow but the item is fine."},
]
buf = io.BytesIO()
for d in documents:
request = {
"custom_id": d["id"],
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "gpt-4.1-mini",
"temperature": 0,
"messages": [
{"role": "system", "content": "Classify sentiment as positive, neutral, or negative. Respond with one word."},
{"role": "user", "content": d["text"]},
],
"max_tokens": 4,
},
}
buf.write((json.dumps(request) + "\n").encode("utf-8"))
buf.seek(0)
input_file = client.files.create(file=("batch.jsonl", buf), purpose="batch")
batch = client.batches.create(
input_file_id=input_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"pipeline": "sentiment-nightly", "run_date": "2026-06-27"},
)
delay = 10
terminal = {"completed", "failed", "expired", "cancelled"}
while True:
batch = client.batches.retrieve(batch.id)
if batch.status in terminal:
break
time.sleep(delay)
delay = min(delay * 2, 120)
if batch.status != "completed":
raise RuntimeError(f"Batch ended in state {batch.status}")
output = client.files.content(batch.output_file_id).text
results = {}
for line in output.splitlines():
row = json.loads(line)
results[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]
for d in documents:
print(d["id"], results.get(d["id"], "MISSING"))
The shape that matters: custom_id is the only reliable way to join output back to input. Output line order is undefined and can differ from input order. (I hit this exact bug on my first batch pipeline, assuming order was preserved, and burned two hours chasing "data corruption" before re-reading the docs.) Always carry your primary key forward as custom_id. Never rely on index or position.
Using the Anthropic Message Batches API in Python
Anthropic's submission shape is more compact. Build an array of request objects, each carrying a custom_id and a full params block matching the Messages API, then post them all in one call. There's no separate file upload step. Results come back as a Server-Sent Events stream of JSON objects.
import time
import anthropic
from anthropic.types.messages.batch_create_params import Request
from anthropic.types.messages import MessageCreateParamsNonStreaming
client = anthropic.Anthropic()
documents = [
{"id": "doc-001", "text": "This product exceeded my expectations."},
{"id": "doc-002", "text": "Shipping was slow but the item is fine."},
]
requests = [
Request(
custom_id=d["id"],
params=MessageCreateParamsNonStreaming(
model="claude-sonnet-4-6",
max_tokens=8,
system="Classify sentiment as positive, neutral, or negative. Respond with one word.",
messages=[{"role": "user", "content": d["text"]}],
),
)
for d in documents
]
batch = client.messages.batches.create(requests=requests)
delay = 10
while batch.processing_status == "in_progress":
time.sleep(delay)
delay = min(delay * 2, 120)
batch = client.messages.batches.retrieve(batch.id)
if batch.processing_status != "ended":
raise RuntimeError(f"Batch ended in state {batch.processing_status}")
results = {}
for result in client.messages.batches.results(batch.id):
if result.result.type == "succeeded":
text = result.result.message.content[0].text
results[result.custom_id] = text
else:
results[result.custom_id] = f"ERROR:{result.result.type}"
for d in documents:
print(d["id"], results.get(d["id"], "MISSING"))
Two things to flag. First, the results() method returns an iterator over the SSE stream. Don't materialize the whole list if your batch is large; stream it row-by-row into your database. Second, individual requests can fail independently (e.g. context-window overflow on one document), and the wrapper indicates failure type per row. Treat per-row failure as expected, not exceptional.
Polling, durable orchestration, and the missing webhook
Neither provider offers a completion webhook at GA. You poll. For ad-hoc work the inline polling loops above are fine. For production, polling needs to live in durable infrastructure: if your worker dies, you must not lose the batch ID, and you must not double-submit the work on restart.
The pattern I run in production: persist the batch ID and the input rows' custom_ids to your application database the instant the batch is created. Then run a separate scheduled worker (cron, Celery beat, or a workflow engine) every 5–15 minutes that picks up open batch records, calls retrieve, and either updates the status or transitions into the download-and-join step. For larger systems, this is the natural home for a durable workflow engine like Temporal, which gives you retries, dead-letter handling, and resumability across worker restarts for free.
Exponential backoff matters even for polling, because polling adds up. A 6-hour batch polled every 10 seconds is 2,160 useless API calls and 2,160 lines of noise in your logs. Start at 10s, double each tick, cap at 120s, and you get roughly the same wakeup latency at the end with about 60 calls total.
Handling partial failures, the error file, and retries
Both providers treat individual request failure as a first-class concept. On OpenAI, the completed batch object exposes an error_file_id alongside output_file_id. The error file is JSONL, one line per failed request, each line carrying the original custom_id and an error code: validation errors (bad model name, missing field), context-length errors, content-filter rejections, or transient backend errors.
The right shape for production: treat the error file as a retry queue, not a log file. After every batch run, parse the error file, classify errors, and decide per-class what to do. Validation errors require a code fix and should page a human. Context-length errors should be retried with a smaller chunked input. Content-filter errors should be skipped and emitted as known-bad. Transient errors should be resubmitted as part of the next batch run.
def reconcile_batch(client, batch):
success_ids, retry_payloads, dead_letter = set(), [], []
if batch.output_file_id:
for line in client.files.content(batch.output_file_id).text.splitlines():
row = json.loads(line)
if row.get("response", {}).get("status_code") == 200:
success_ids.add(row["custom_id"])
if batch.error_file_id:
for line in client.files.content(batch.error_file_id).text.splitlines():
row = json.loads(line)
code = row.get("error", {}).get("code", "")
if code in ("rate_limit_exceeded", "server_error"):
retry_payloads.append(row["custom_id"])
else:
dead_letter.append((row["custom_id"], code))
return success_ids, retry_payloads, dead_letter
One detail that catches teams: a batch can finish with zero successful requests and still report status: completed. "Completed" means the job finished running, not that any of your work succeeded. So always check the output and error files, and gate downstream pipeline steps on a minimum success rate, not on batch status alone. Pair this with LLM observability tracing so each batch row stays recoverable to its source prompt for debugging.
Batch limits, tier quotas, and the constraints nobody mentions
The per-batch limits are well-documented: OpenAI 50,000 requests or 200 MB; Anthropic 100,000 requests. The limits that actually bite are the account-level queue limits, which are tier-dependent and only show up when you exceed them.
OpenAI enforces a "batch queue limit" measured in queued tokens per model, separate from your synchronous TPM and RPM. On tier 1 it's commonly 90K queued tokens for GPT-4.1; on tier 5 it climbs to 5B. Your tier is based on cumulative spend over time. The error you'll see is batch_queue_limit_exceeded, and the fix is either to wait for in-flight batches to drain, split work across more days, or contact sales for a tier increase. You won't see this limit until you hit it.
Anthropic enforces concurrent-batch caps per account, typically 10 concurrent batches on standard tiers with higher limits for enterprise. Combined with the 100,000-request batch cap, that's effectively a 1M-request-in-flight ceiling without an upgrade.
Cost modeling: when does batch save real money?
The 50% headline number is correct, but it undersells the actual win when you combine batch with caching. Anthropic applies the batch discount on top of prompt caching: a 10K-token system prompt cached at 90% off, then run through batch at 50% off, lands at roughly 5% of the headline input price. On Sonnet 4.6 ($3.00 input), that's $0.15 per million cached input tokens. For workloads with stable, long system prompts (classification, extraction, evaluation), this stacking is the single largest optimization available short of switching models.
OpenAI's 2026 model lineup with batch is similarly compressed at the low end. GPT-4.1 Nano lands at $0.05/$0.20 per million tokens under batch, which makes large-scale classification and tagging effectively free relative to the cost of the rest of your pipeline. GPT-4.1 batches at $1.00/$4.00, competitive with synchronous Sonnet 4.6 for general-purpose tasks. Match your model to the task: cheap models for high-volume narrow work, flagship for genuinely hard reasoning.
A back-of-envelope rule that's held up across the systems I've built: any workflow that spends more than $500/month synchronously is a batch candidate. Below that, the engineering time costs more than the savings. Above $5,000/month, batching isn't optional, it's the difference between "this product is profitable" and "this product loses money on every customer." Pair batch with a strict structured-output schema so downstream parsing of 50,000 returns is deterministic, not best-effort.
Can I use streaming responses with the OpenAI or Anthropic batch APIs?
No. Both batch APIs are single-shot, non-streaming. Requests inside a batch must complete fully before any output is returned. If you need streaming, use the synchronous endpoints. Batch is the wrong tool for any latency-sensitive path.
Are batch API results lower quality than synchronous responses?
No. Both providers route batch traffic to the same model weights as their synchronous endpoints. The only difference is when the result lands, not what's in it. Same model, same temperature, same output distribution, at half the price.
What happens if my batch expires before completing?
Both providers cancel any remaining work and return whatever completed successfully via the output file. You're billed only for completed requests. Plan for partial completion by joining results on custom_id and resubmitting unmatched rows in the next batch run.
Can I use tool calling and function calling inside a batch request?
You can declare tools in the request body and receive a single tool-call response, but multi-turn tool execution loops aren't supported inside a batch. If your workflow requires the model to call a tool, see the result, and decide again, run that loop synchronously and put only the final single-shot generation into a batch.
How do I know which model tier supports the batch API?
On OpenAI, all current chat-completion models and the text-embedding-3 family support batch. On Anthropic, all GA Claude models (Haiku 4.5, Sonnet 4.6, Opus 4.7, Opus 4.8) support Message Batches as of mid-2026. Preview and beta models are typically batch-eligible after they hit GA. Check the model card before relying on it.
Voyage-3-large tops MTEB v2, Cohere Embed v4 handles 128K multimodal, OpenAI is the best default, and Jina wins multilingual. Full 2026 benchmark with code.
When to pick structured outputs vs function calling across OpenAI, Anthropic, and Gemini in 2026: schema enforcement, latency, and runnable code for each pattern.
A production guide to prompt caching with Claude, OpenAI, and Gemini. Learn cache breakpoints, TTL strategy, prompt structure, and the seven mistakes that silently kill your cache hit rate.
