Semantic Router
Every request goes to the model that should answer it.
Some questions need your most capable model. Most do not. The router reads each one, sends it where it belongs, and shows you what it cost.
The problem
One model is answering everything.
Whichever model your applications were pointed at is the model that handles every prompt, from a one-line lookup to your hardest question of the year.
What happens when that provider throttles you?
Everything behind it slows down at the same time.
Which model answered this?
Nobody can tell you without reading the code.
What if another model is better at this?
Trying it means changing the application first.
How it works
The question decides where it goes.
A question arrives. What answers it is decided then, rather than fixed in advance.
Someone asks
“Read this invoice and return the PO number.”
Nightshift Semantic Router
Reads the question and picks the model for it.
- A small model in your environmentlookups, extraction, the simple majority
Your enterprise platformeveryday work, and anything needing tools
A frontier modelthe hardest questions you have
The answer comes back the same way, and nothing in your systems has to change.
Balancing
It chooses a model on every request.
Not once, when your systems were built. Every time, and from whatever is healthy at that moment. The router has no favorite, and the provider you already use is one of the places it can send a request.
Scores what matters
Quality, price and live speed, weighed for the prompt in front of it.
Drops what is struggling
A model that is throttled or slow stops receiving traffic until it recovers.
Spends your commitment first
Traffic goes to the capacity you already pay for before anything metered by the token.
Nobody notices
A provider having a bad afternoon becomes a different model answering, rather than an error someone has to explain.
Each of these can hold as many models as you like.
Models in your environment
running inside your own environment
Capacity you control completely, and the only one nothing leaves.
Capacity you control
The provider you standardized on
the agreement you already negotiated
Capacity you are paying for whether or not you use it this month, so it gets used first.
Used first, never replaced
Frontier providers
over their APIs, as many as you keep keys for
Your most capable models, and somewhere to go when the rest is full.
Capacity to burst into
Tokenomics
Get the full picture.
Every call is recorded against the workflow that made it and the model that served it. Token use and spend across every provider on one screen, while there is still time to act on it.
Tokens tracked across every model
One place for traffic and spend you currently read across separate invoices.
Attributed to a workflow
Your invoices arrive by account. This arrives by the workflow that spent it.
Ahead of the bill
You see a workflow getting more expensive with a month left to change it.
Governance
Some prompts should never leave.
The same reading that picks a model also rules models out. A prompt carrying personal data, or covered by a rule about where data may go, never reaches a provider that would break it.
Personal data never leaves your network.
A prompt carrying it is held on a model inside your own perimeter.
A jailbreak never reaches a provider.
Refused at the router, and the attempt is on the record.
Every decision can be reopened.
What was read, what it matched, and which model answered it.
What changes
96%
lower effective cost
Where routing let a small model recover most of a frontier model’s performance on a task.
Published by the vLLM Semantic Router project
86%
answered without a metered call
Share of prompts served by a free self-hosted model in the same published test.
Published by the vLLM Semantic Router project
3 of 3
benchmarks matched or beaten
A routed mixture against the single strongest model available, so the saving is not a quality trade.
Published by the vLLM Semantic Router project
What we deliver
We deploy it and tune it around your workflows.
You get the deployment, the policies that decide where each request goes, and the evidence that those policies are right.
Deployed in your cloud
It runs in your own cloud account, alongside the models it routes to.
We connect what you already run
Starting with the provider you are committed to, then whatever you want beside it.
Policies built with you
We start by watching, then write the rules once the records show what each workflow needs.
Standing evals
Every routed workflow keeps being scored, so a routing decision stays defensible after we leave.
FAQs
No. Capacity you already pay for is the cheapest capacity you have, so the router sends traffic there first and anything else sits alongside it. No application changes, and neither does your agreement.
No. Nothing in your systems has to be rebuilt or repointed. Which model answers is settings inside the router, so adding one is a change we make, not a project you run.
It runs in your own cloud with more than one instance behind health checks, and with a fallback we agree with you before it goes live. That fallback is the setup you run today, so a bad day for the router is a normal day for everyone else.
About 40 milliseconds at the median, measured by the router project on CPU with no dedicated GPU. That is 0.4 to 5% of the time the model itself takes.
It reads the prompt, rules out any model your policy does not allow for it, then scores what is left on quality, price and live speed. A slow or throttled model is not eligible until it recovers.
Nothing moves to a different model until the evals show it scores the same on that workflow’s own traffic. Where it does not, the workflow keeps the model it has.
Only where you allow it to. The router runs in your own cloud account, and a prompt carrying personal data or covered by a residency rule is held on a model inside your network.
Models you host yourself, whatever your cloud agreements include, and as many external providers as you keep keys for.
The engine is the open source vLLM Semantic Router and it keeps running. The policies are configuration in your own repository, so you keep those too.
See where your model spend actually goes.
Start by watching. It records every request and routes none of them, so you begin with a picture of what you already run.