Semantic Routing for LLM Apps: Route Queries by Intent (2026)

Updated: September 15, 2026

Semantic routing for LLM apps is the practice of sending a user's query to a specific handler (a tool, prompt, agent, or model) by comparing the query's embedding to a set of pre-computed intent examples, rather than asking another LLM to decide. It replaces a "router prompt" round-trip with a single vector similarity lookup, cutting p50 routing latency from 400–900 ms to under 40 ms while keeping decisions auditable. Honestly, I've swapped GPT-4o "which tool do I use?" prompts for semantic routers on three production stacks in the last year, and the shape of the win is always the same: cheaper, faster, and dramatically easier to debug.

• Semantic routing matches the user query's embedding against pre-defined route "utterances" and picks the closest match. No LLM call required for the routing decision itself.
• It's typically 10–30x cheaper and 20–50x faster than LLM-based routing because it uses a single embedding call and cosine similarity instead of a chat completion.
• The Aurelio Labs semantic-router library dominates the open-source space; commercial alternatives include Portkey Conditional Routing and LangChain's RouterChain.
• Use semantic routing when your intent space is bounded (fewer than ~50 routes) and examples are cheap to write. Fall back to LLM routing for open-ended tasks.
• Always cache embeddings, calibrate a similarity threshold on real user queries, and add a "no-match" route that escalates to LLM classification.
• Semantic routing composes naturally with API gateways (route by intent first, then by model preference) instead of replacing them.

What is semantic routing in LLM apps?

Semantic routing is the routing layer of an LLM application that decides, per incoming request, which downstream component should handle the query (a specific tool, prompt template, retrieval index, agent, or even a different model) by embedding the query and comparing it to a small library of labelled example utterances. The route with the highest cosine similarity above a configured threshold wins.

Think of it as an intent classifier that requires no training. You write 5–15 example utterances per route, embed them once at startup, and every subsequent decision is a dot product. If the top score doesn't clear the threshold, the router either returns "no match" or falls through to an LLM classifier as a safety net.

This matters because a surprising amount of what people call "agent reasoning" is really "route to the right sub-agent." When I audit an agent pipeline that feels sluggish, the first thing I look at is whether a router prompt is running before every tool call. Nine times out of ten, that prompt is doing shallow keyword matching that a bag of embeddings can do in milliseconds. Semantic routing pulls that decision out of the LLM's job description and into infrastructure, where it belongs.

The technique has been standard in traditional NLP for years. It's essentially nearest-neighbour intent classification. What's new is that pretrained embedding models like text-embedding-3-large, Cohere's embed-v4, and Voyage voyage-3 ship semantic quality that beats fine-tuned BERT classifiers on most real intent sets, without any training data beyond a handful of examples.

How does semantic routing work under the hood?

A semantic router has three moving parts: a set of routes, an encoder that produces embeddings, and a decision function that picks the winning route.

Routes and utterances

Each route is a name plus a list of example phrases (utterances) that a user might say when they want that route. For a customer support bot, a "billing" route might have utterances like "why was I charged twice", "cancel my subscription", and "refund policy". Utterances are not templates. They are literal example phrases, and the router doesn't parse them.

The encoder

The encoder turns strings into fixed-dimension vectors. In 2026 the practical choices are OpenAI's text-embedding-3-small (1536 dims, $0.02/M tokens, best latency-per-dollar for English), Cohere embed-v4 (multilingual, 1024 dims), Voyage voyage-3 (highest MTEB score for retrieval), or a local sentence-transformers model like all-MiniLM-L6-v2 for privacy-sensitive deployments. Whichever you pick, both the utterances and the query must go through the same encoder. Mixing encoders silently poisons your similarity scores.

The decision function

At request time, the router embeds the incoming query, computes cosine similarity against every stored utterance, and aggregates per-route (usually by taking the max similarity, but mean or top-k mean also work). The route with the highest aggregate score wins if it clears a threshold, commonly 0.75 for OpenAI embeddings, but this must be calibrated against real traffic.

# The decision function in ~10 lines
import numpy as np

def route(query_embedding, route_index, threshold=0.75):
    scores = {}
    for route_name, utterance_embeddings in route_index.items():
        # cosine similarity, embeddings assumed L2-normalized
        sims = utterance_embeddings @ query_embedding
        scores[route_name] = sims.max()
    best_route, best_score = max(scores.items(), key=lambda kv: kv[1])
    return best_route if best_score >= threshold else None

That's the whole trick. Everything after (hierarchical routes, per-route thresholds, dynamic route loading, LLM fallback) is scaffolding around this ten-line kernel.

Semantic Router vs LLM Router vs rules: which should you use?

The three viable approaches to routing in a modern LLM app are (1) hard-coded rules or regexes, (2) semantic routing via embeddings, and (3) delegating the decision to an LLM ("LLM Router" or a router prompt). Each has a natural home.

Dimension	Rules / Regex	Semantic Router	LLM Router
Latency (p50)	< 1 ms	20–40 ms	300–900 ms
Cost per decision	Free	~$0.00002 (1 embed call)	~$0.001–0.005 (1 chat call)
Handles paraphrase	No	Yes	Yes
Handles novel intents	No	Poorly (needs new utterances)	Yes
Auditability	Very high	High (similarity scores logged)	Low (natural-language reasoning)
Effort to add a route	Write regex	Add 5–15 utterances	Update system prompt
Good for	Slash commands, keywords	Bounded intent set, 5–50 routes	Open-ended tasks, agent tool selection

In practice these compose. My default architecture layers them: rules catch the trivial cases (slash commands, admin flags), a semantic router handles 80% of user intents that map to a known set of skills, and an LLM classifier catches the long tail. Each layer only fires if the previous one returned "no match," so the average query pays for maybe one embedding call and never touches a chat completion for routing.

This is the same failover discipline I described in the article on LLM circuit breakers and fallback chains, applied at the routing layer instead of the provider layer.

How to build a semantic router in Python (step by step)

The reference implementation is aurelio-labs/semantic-router, Aurelio Labs' open-source library, MIT licensed, maintained by James Briggs. It handles route management, encoder abstraction, thresholds, dynamic routes, and hybrid routing (adding a sparse BM25 signal to dense embeddings). Here's a complete example you can paste and run.

1. Install and import

pip install "semantic-router[openai]==0.1.5"

2. Define your routes

import os
from semantic_router import Route, RouteLayer
from semantic_router.encoders import OpenAIEncoder

# Each route is a bounded intent with 5-15 example utterances.
billing = Route(
    name="billing",
    utterances=[
        "why was I charged twice this month",
        "cancel my subscription",
        "I want a refund",
        "how do I update my card",
        "my invoice is wrong",
        "downgrade my plan",
        "when will I be billed next",
    ],
)

technical_support = Route(
    name="technical_support",
    utterances=[
        "the app crashes when I click export",
        "I can't log in",
        "why is the dashboard showing zero",
        "my webhook stopped firing",
        "the api returns 500",
        "reset my password",
    ],
)

sales = Route(
    name="sales",
    utterances=[
        "do you offer enterprise plans",
        "can I get a demo",
        "what's your pricing for teams",
        "is there a discount for annual billing",
        "who do I talk to about a custom contract",
    ],
)

routes = [billing, technical_support, sales]

3. Build the router

encoder = OpenAIEncoder(
    name="text-embedding-3-small",
    openai_api_key=os.environ["OPENAI_API_KEY"],
)

# score_threshold: minimum cosine similarity for a match.
# 0.75 is a decent default for text-embedding-3-small; calibrate on real traffic.
router = RouteLayer(
    encoder=encoder,
    routes=routes,
    score_threshold=0.75,
)

4. Route incoming queries

decision = router("my card was declined but I got charged anyway")
# decision.name == "billing", decision.function_call == None

if decision.name is None:
    # No route cleared the threshold, fall back to LLM classification
    decision_name = llm_classify(query)
else:
    decision_name = decision.name

handler = HANDLERS[decision_name]
response = handler(query)

5. Persist and hot-reload

In production, you want to add and remove routes without redeploying. Semantic Router supports remote route stores backed by Redis or PostgreSQL. The router polls for changes and rebuilds the in-memory index. This is how support teams add new intents on the fly as they see novel queries in the "no match" bucket.

Handling multi-intent queries and route hierarchies

Real users send messages that carry more than one intent: "cancel my sub and refund last month's charge." Or they send hierarchical intents that split by department first and then by action: billing → refund vs billing → payment method update.

Multi-intent via top-k

The trick is to fire not just the top route but every route whose score clears the threshold. Semantic Router exposes this through RouteLayer.retrieve_top_routes(). Your downstream handler then decides whether to run them sequentially, in parallel, or ask a clarifying question. In my agent stacks I run at most two concurrent routes and only if both scores are within 0.05 of each other; below that gap I trust the top match.

Hierarchical routing

For hierarchies, chain two routers. The first router picks the coarse category (billing, tech support, sales); the second router, chosen dynamically based on the first result, picks the fine-grained action. This keeps each router's utterance list small, which improves both speed and accuracy. One large router with 200 utterances across 40 mixed intents ends up with tangled decision boundaries.

coarse = coarse_router(query)  # e.g. "billing"
if coarse.name:
    fine = FINE_ROUTERS[coarse.name](query)  # billing_router picks "refund" vs "cancel"
    return DISPATCH[coarse.name][fine.name](query)

This is the same shape as a multi-agent orchestration pattern, where a router-agent delegates to worker-agents, except the routing decision is a cosine similarity, not a chat completion.

Production considerations: thresholds, caching, and drift

Calibrate thresholds on real queries, not vibes

Every embedding model has its own similarity distribution. The 0.75 default for text-embedding-3-small becomes 0.55 for MiniLM-L6, 0.68 for Voyage-3, and 0.82 for Cohere embed-v4. The only reliable way to set a threshold is to log query-plus-scores from 500–1000 real production queries, hand-label the correct route on a sample, and pick the threshold that maximises F1 (or precision if false positives are worse than false negatives, which is usually true for billing/refund flows).

Cache embeddings aggressively

The utterance embeddings never change unless you edit the route definitions. Store them in Redis or a local file and only re-embed on route changes. For query embeddings, add a small LRU cache keyed by the normalised query. Repeat queries are common enough that even a 10k-entry cache pays for itself within a day. For the deeper caching story I wrote up in LLM cost optimization, semantic caching at the router layer is one of the cheapest wins available.

Detect drift with a "no match" queue

Every query that fails the threshold goes to a queue. Once a week (or nightly if you're moving fast), sample from that queue, cluster the embeddings, and manually label the clusters. New user intents show up here as tight clusters before they become 5% of your traffic. This is your early-warning system for when to add a new route. Skipping this step is how routers silently degrade: intent distributions drift, threshold-clearing rate creeps down, and one day your "no match" bucket is 30% of traffic and everyone's mad.

Choose your encoder for latency, not just quality

OpenAI's embedding endpoints add 60–120 ms of network latency. If routing is on the hot path, that dominates the entire router budget. For latency-critical apps, run a small local model. The all-MiniLM-L6-v2 model from sentence-transformers is 384 dims, sub-10 ms on CPU, and adequate for 90% of intent classification tasks. Save the big-cloud encoder for the utterance embeddings you compute once at startup.

Common pitfalls and how to avoid them

Overloaded routes

A "general" or "other" route with 40 utterances covering every possible fallback will absorb queries it shouldn't, because its utterance surface area is huge. Delete it. Use "no match" plus LLM fallback instead. Semantic routing is a positive-selection tool, not a garbage collector.

Utterances written by developers, not users

Developers write utterances the way documentation would: "reset password." Users write "I forgot my login." The utterance set should be sampled from real user messages, not made up at the keyboard. Once you have production data, replace synthetic utterances with real ones from the "no match" bucket. This alone will typically raise accuracy 5–15 percentage points.

Ignoring language and casing

Embedding models are casing-tolerant but not always accent-tolerant, and multilingual quality varies. If your users type in more than one language, use a multilingual encoder from the start (Cohere embed-v4 or paraphrase-multilingual-MiniLM-L12-v2). Retro-fitting multilingual support later means re-embedding your whole utterance index and re-calibrating thresholds.

Skipping the fallback

Semantic routing is closed-set by definition. It can only pick routes you've defined. Every production system needs an escape hatch: a "no match" event that either escalates to a human, defers to an LLM classifier, or returns a graceful "I'm not sure what you mean, can you rephrase?" prompt. I hit this exact bug shipping a support bot last spring, where a router silently fell through to a default handler and dispatched the wrong workflow for hours before anyone noticed. Always alarm on the "no match" rate.

Routing where you should be retrieving

If your "routes" are really just different knowledge domains and the handler is the same LLM with different context, you don't need a router. You need retrieval. Semantic routing shines when the handlers are structurally different (a Python function, a tool call, a specialised prompt, a different model). If they're all "call the LLM with different docs," a well-tuned RAG pipeline with metadata filtering does the job with less scaffolding.

Frequently Asked Questions

Do I need training data to use semantic routing?

No. Semantic routing uses pretrained embedding models, so you only need 5–15 example utterances per route. There's no gradient descent, no training loop, and no held-out validation set. You write examples, embed them once, and the router is live. Traditional intent classifiers required hundreds of labelled examples per intent; embedding-based routing removes that data requirement almost entirely.

How is semantic routing different from RAG?

RAG retrieves document chunks to inject into a prompt; semantic routing picks which handler runs. The mechanics are similar (both embed the query and rank candidates by cosine similarity) but the outputs and consumers differ. RAG returns text passages for an LLM to read; a semantic router returns a route name that dispatches to code. Many production stacks use both: the router picks the tool, and RAG feeds it context once dispatched.

When should I use an LLM Router instead of semantic routing?

Use an LLM router when the intent space is unbounded or when routing requires reasoning about the user's goal beyond surface similarity, for example agent tool selection where the model needs to decide whether a multi-step plan is required. Semantic routing wins on bounded, well-enumerated intent sets where latency and cost matter. In practice, use both: semantic router first, LLM router as fallback.

What similarity threshold should I use?

It depends on your encoder. For OpenAI text-embedding-3-small, start at 0.75; for Cohere embed-v4, start at 0.82; for local MiniLM-L6, start at 0.55. Then calibrate by logging real query scores, hand-labelling a sample of 500 queries, and picking the threshold that maximises F1, or precision if false-positive routing has expensive consequences.

Can semantic routing handle multiple intents in one query?

Yes, by taking the top-k routes above the threshold rather than only the top-1. If two scores are within roughly 0.05 of each other, treat the query as multi-intent and either dispatch both handlers in parallel or ask a clarifying question. For deeply nested cases (several intents plus ordering constraints), an LLM router does a cleaner job than trying to force semantic routing to handle everything.