Parallel Tool Calls in Production: OpenAI, Anthropic, and Gemini Function Calling Compared (2026)

Updated: July 26, 2026

Parallel tool calls let a single LLM response request multiple tool invocations that you execute concurrently and return in one follow-up message, cutting agent latency by 40 to 70% on multi-step tasks. In 2026, OpenAI, Anthropic, and Google Gemini all support parallel function calling, but the request format, opt-out flags, and result-passing semantics differ enough that a working OpenAI implementation will silently break on Anthropic. So, let's walk through the three APIs side-by-side, look at runnable Python for each, and figure out when it's smarter to force serial execution instead.

• OpenAI enables parallel tool calls by default. Disable with parallel_tool_calls=false when using strict Structured Outputs or when tools have ordering dependencies.
• Anthropic's Claude 4/4.5 models emit multiple tool_use blocks in one assistant turn, and you must return every corresponding tool_result in one user message keyed by tool_use_id.
• Gemini 2.5 supports "compositional function calling" (sequential dependent calls in one turn) plus true parallel calls via multiple functionCall parts.
• Execute tools with asyncio.gather or a bounded semaphore. Sequential await defeats the whole point of parallelism.
• Always evaluate parallel vs serial on your task suite. Parallel wins on latency but occasionally reduces reasoning quality on tightly coupled tool chains.
• Use an LLM gateway to normalize vendor differences if you route across all three providers.

What are parallel tool calls?

A parallel tool call happens when the model, in a single assistant turn, requests two or more independent tools at once (for example, get_weather("Tokyo") and get_weather("Osaka")) so your runtime can dispatch them in parallel rather than looping the model through two sequential round-trips. Every extra round-trip costs a full network hop plus prefill and decode time, so shaving three round-trips off a five-step agent typically drops wall-clock latency from 8 to 12 seconds down to 2 to 4.

The pattern relies on the model recognizing that two calls have no data dependency between them. If step B needs the output of step A, the model should (and usually does) emit only step A first, then produce step B in the next turn. In my experience testing across the three major vendors, GPT-4.1 and Claude Sonnet 4.5 make this judgment reliably. Smaller models like GPT-4o-mini and Claude Haiku 4.5 occasionally batch calls that shouldn't be batched, which is why evaluation harnesses matter more once you flip parallelism on.

One thing worth flagging: "parallel" here refers to the request. The model emits N tool call requests in one response. Actually executing them concurrently is your job on the server side, and skipping the concurrency implementation is the single most common reason teams see zero latency improvement after enabling the feature. (I've reviewed at least three codebases where the team was convinced parallel was broken, and it turned out they were awaiting each call in a for loop.)

Vendor comparison at a glance

Before the code, here's the semantic and API-shape comparison across the three providers as of July 2026:

Feature	OpenAI (GPT-4.1, GPT-5)	Anthropic (Claude 4/4.5)	Google (Gemini 2.5)
Default parallel behavior	Enabled	Enabled	Enabled
Opt-out flag	parallel_tool_calls=false	disable_parallel_tool_use=true (in tool_choice)	function_calling_config.mode="ANY" with single allowed name
Response shape	message.tool_calls array	Multiple tool_use blocks in content	Multiple functionCall parts
Result linking key	tool_call_id	tool_use_id	Function name (positional order)
Result message role	tool (one per call)	user (all results in one message)	user / function role parts
Strict schema + parallel	Mutually exclusive	Compatible	Compatible
Compositional (dependent) calls in one turn	No	No	Yes (2.5 feature)

The most consequential row is the last two. OpenAI forbids combining strict: true Structured Outputs with parallel calls; the model falls back to non-strict schema when parallel is on. If you rely on 100% schema conformance, you either turn parallel off or validate outputs post-hoc with something like Instructor or Pydantic. Anthropic and Google don't have this restriction.

OpenAI parallel function calling

OpenAI's Chat Completions and Responses APIs return a tool_calls array on the assistant message. Each entry has a unique id, a type, and a function object with name and arguments (a JSON-encoded string, not a parsed object, which catches many first-time users). You then send one message per call back with role tool and matching tool_call_id.

from openai import AsyncOpenAI
import asyncio, json

client = AsyncOpenAI()

TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

async def get_weather(city: str) -> dict:
    await asyncio.sleep(0.5)  # simulate network I/O
    return {"city": city, "temp_c": 24, "condition": "clear"}

async def run():
    messages = [{"role": "user", "content": "Compare weather in Tokyo, Osaka, and Kyoto."}]

    resp = await client.chat.completions.create(
        model="gpt-4.1",
        messages=messages,
        tools=TOOLS,
        parallel_tool_calls=True,  # default; shown for clarity
    )
    msg = resp.choices[0].message
    messages.append(msg.model_dump(exclude_none=True))

    # Dispatch all tool calls concurrently
    async def dispatch(call):
        args = json.loads(call.function.arguments)
        result = await get_weather(**args)
        return {
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        }

    results = await asyncio.gather(*(dispatch(c) for c in msg.tool_calls))
    messages.extend(results)

    final = await client.chat.completions.create(model="gpt-4.1", messages=messages)
    print(final.choices[0].message.content)

asyncio.run(run())

Anthropic parallel tool use

Anthropic's Messages API returns multiple tool_use content blocks in the assistant's content array whenever the model decides to fan out. The critical rule that trips people up: you must return every corresponding tool_result in a single user message, each keyed by the exact tool_use_id. If you return them across multiple user messages, or omit one, the API throws a 400 error. (I hit this exact bug shipping my first Claude agent, and it took an embarrassing amount of debugging.)

from anthropic import AsyncAnthropic
import asyncio

client = AsyncAnthropic()

TOOLS = [{
    "name": "get_weather",
    "description": "Get current weather for a city",
    "input_schema": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}]

async def get_weather(city: str) -> dict:
    await asyncio.sleep(0.5)
    return {"city": city, "temp_c": 24, "condition": "clear"}

async def run():
    messages = [{"role": "user", "content": "Compare weather in Tokyo, Osaka, and Kyoto."}]

    resp = await client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        tools=TOOLS,
        messages=messages,
    )
    messages.append({"role": "assistant", "content": resp.content})

    tool_uses = [b for b in resp.content if b.type == "tool_use"]

    async def dispatch(block):
        result = await get_weather(**block.input)
        return {
            "type": "tool_result",
            "tool_use_id": block.id,
            "content": str(result),
        }

    results = await asyncio.gather(*(dispatch(b) for b in tool_uses))
    # All results in ONE user message
    messages.append({"role": "user", "content": results})

    final = await client.messages.create(
        model="claude-sonnet-4-5", max_tokens=1024, tools=TOOLS, messages=messages,
    )
    print(final.content[0].text)

asyncio.run(run())

To force serial execution, pass tool_choice={"type": "auto", "disable_parallel_tool_use": True}. See the Anthropic tool use documentation for the full schema. Note that disable_parallel_tool_use works with auto, any, and tool choice modes.

Gemini parallel function calling

Gemini 2.5's response can contain multiple functionCall parts in a single Content, and (uniquely among the three) it also supports "compositional function calling" where the model chains dependent calls within one generation. In practice, that means Gemini can sometimes complete in one turn what OpenAI and Anthropic need two turns for. For the pure parallel-independent-calls case though, the shape is straightforward.

from google import genai
from google.genai import types
import asyncio

client = genai.Client()

def get_weather(city: str) -> dict:
    return {"city": city, "temp_c": 24, "condition": "clear"}

weather_tool = types.Tool(function_declarations=[{
    "name": "get_weather",
    "description": "Get current weather for a city",
    "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}])

async def run():
    contents = [types.Content(role="user", parts=[
        types.Part(text="Compare weather in Tokyo, Osaka, and Kyoto.")
    ])]

    resp = await client.aio.models.generate_content(
        model="gemini-2.5-pro",
        contents=contents,
        config=types.GenerateContentConfig(tools=[weather_tool]),
    )
    contents.append(resp.candidates[0].content)

    calls = [p.function_call for p in resp.candidates[0].content.parts if p.function_call]

    async def dispatch(fc):
        result = get_weather(**dict(fc.args))
        return types.Part.from_function_response(name=fc.name, response=result)

    parts = await asyncio.gather(*(dispatch(fc) for fc in calls))
    contents.append(types.Content(role="user", parts=parts))

    final = await client.aio.models.generate_content(
        model="gemini-2.5-pro",
        contents=contents,
        config=types.GenerateContentConfig(tools=[weather_tool]),
    )
    print(final.text)

asyncio.run(run())

Refer to the Gemini function calling documentation for the compositional-call mode and the FunctionCallingConfig options that let you force a specific tool or disable calling entirely.

Executing tools concurrently in Python

Emitting N parallel tool calls buys you nothing if you execute them one at a time. The right pattern is asyncio.gather with a bounded semaphore. The semaphore caps how many concurrent calls you fire against your own downstream services, which prevents a chatty agent from taking out its own database.

import asyncio
from typing import Callable, Awaitable

SEM = asyncio.Semaphore(8)  # tune per downstream capacity

async def bounded_dispatch(fn: Callable[..., Awaitable], **kwargs):
    async with SEM:
        try:
            return await asyncio.wait_for(fn(**kwargs), timeout=15)
        except asyncio.TimeoutError:
            return {"error": "tool_timeout"}
        except Exception as e:
            return {"error": f"tool_failed: {type(e).__name__}: {e}"}

# Usage inside your tool loop:
async def run_tool_calls(calls, tools_by_name):
    async def one(call):
        fn = tools_by_name[call.name]
        return await bounded_dispatch(fn, **call.args)
    return await asyncio.gather(*(one(c) for c in calls))

Three practical rules I apply on every agent I ship. First, always wrap the tool call in wait_for so a hung tool doesn't stall the whole batch. Second, always return an error object rather than raising, because the model needs to see the failure to recover. Third, always bound concurrency by the slowest downstream, not by the LLM's willingness to fan out. See our production function calling guide for the full retry-and-backoff pattern.

When to disable parallel tool calls

Parallel isn't free. Turn it off in these five specific cases:

• Ordered side effects. If tool B mutates state that tool A read, parallel execution creates a race. The classic example is read_balance then transfer_funds. Never let the model batch these.
• Strict Structured Outputs on OpenAI. Setting strict: true on a function schema with parallel_tool_calls=true silently degrades to non-strict. Pick one.
• Idempotency-sensitive external APIs. Payment providers, email sends, and other services that don't handle duplicate concurrent requests well.
• Debugging. When your evals are failing and you don't know why, serialize temporarily. Parallel makes traces harder to reason about.
• Cost-optimized batch jobs. If you're using the Batch API for 50% cheaper inference, latency doesn't matter and parallel calls just muddy the trace.

Evaluating parallel vs serial in production

Before flipping parallelism on for a live agent, run a paired eval. I keep a fixture set of 50 to 200 real user tasks, run them twice (once serial, once parallel), and score four dimensions: task success rate, wall-clock p50 and p95, number of tool calls made, and cost. The interesting outcome isn't "parallel is faster" (it always is on latency). It's whether task success rate holds. On roughly 10% of the agents I've measured, parallel drops success by 2 to 5 points because the model over-batches and misses a dependency.

The pragmatic answer is often mixed-mode: parallel on for read-only tools (search, fetch, lookup) and serial forced for write tools. Anthropic and Gemini let you group tools with different tool_choice settings across turns; OpenAI is coarser and forces you to toggle at the request level. That vendor difference matters if you're routing across providers with a LiteLLM-style gateway. Normalize the semantics upstream or your eval numbers won't be comparable.

Common failure modes and fixes

Missing tool_result for one of the tool_use blocks (Anthropic)

You'll see a 400 like "tool_use ids were found without tool_result blocks". Cause: you filtered out one of the calls, or one of your dispatch tasks raised and you didn't handle it. Fix: always return a placeholder tool_result with an error payload for failed dispatches. The API needs the id round-trip; the model handles the error message fine.

arguments field is a string, not a dict (OpenAI)

Passing call.function.arguments directly into **kwargs gives you a TypeError: argument of type 'str' is not iterable. Always json.loads first, and wrap in a try/except. Occasionally the model emits invalid JSON, especially with parallel calls where token budget is tighter.

Silent quality regression after enabling parallel

Symptom: latency drops as expected, but a slice of your evals starts failing. Cause is almost always that two tools you thought were independent share hidden state (a cache, a session, a user preference read). Fix: audit which tools mutate shared resources and force those into a serial group by omitting the parallel flag on turns where those tools are candidates.

Rate-limit storms under real traffic

Parallel tool calls multiply the load on downstream services by whatever the average fan-out is (typically 2 to 4x). If your downstream can't handle it, cap the semaphore lower than you think you need to and add jittered retry. See the prompt caching guide for backoff patterns that also apply here.

Frequently Asked Questions

Do all LLMs support parallel function calling?

The three major closed-model vendors (OpenAI's GPT-4.1 and GPT-5, Anthropic's Claude 4 and 4.5, and Google's Gemini 2.5) support parallel tool calls as of July 2026. Open-weight models like Llama 3.3 and Qwen 3 support it through inference frameworks (vLLM, TGI) but with less consistent behavior. Test on your task suite before assuming parity.

How do you disable parallel tool calls on Claude?

Pass tool_choice={"type": "auto", "disable_parallel_tool_use": true} in the Messages API request. This forces the model to emit at most one tool_use block per turn. It works with type: "any" and type: "tool" as well.

Why does OpenAI disable strict schema when parallel_tool_calls is on?

Strict Structured Outputs uses constrained decoding, which currently doesn't compose with the sampling path that produces multiple tool calls in one response. If you need both guaranteed schema conformance and low latency, either turn parallel off, or leave parallel on and validate arguments post-hoc with Pydantic, accepting the occasional invalid call as a retry cost.

Can Gemini call two tools that depend on each other in one turn?

Yes. Gemini 2.5 supports compositional function calling, where the model can generate a dependent chain (call A, use A's output as input to B) inside a single response. OpenAI and Anthropic can't do this; they require a round-trip after each dependent call. In practice, compositional calling is useful for read-then-transform pipelines but adds latency inside the generation.

Does parallel tool calling cost more tokens?

Slightly. The assistant response holds N tool call blocks instead of one, and the follow-up user message carries N results. Net token overhead is usually 5 to 15% versus serial, but you save an entire prefill for each round-trip you eliminate, so cost typically drops per completed task even as per-response tokens rise.