Infrastructure
Rate limiting an LLM gateway, when the attacker is your own engineer's agent
A gateway that routes prompts to cheaper models is a cost control. It is also the most expensive open port you will ever run, and the first thing to overload it is usually a coding agent retrying in a loop with a token you issued.
A gateway that routes prompts to cheaper models is a cost control. It is also the most expensive open port you will ever run, and the first thing to overload it is usually an engineer's coding agent, retrying in a loop, holding a token you issued.
That single failure mode changes where the rate limiter goes, what it counts, and who it is scoped to. Most of the defaults are wrong. This piece is about the shape of the problem, the placement that follows from it, and what to count once it is in place.
Why denial of service works at all
A denial of service attack is a flood of requests aimed at a system until it can no longer serve the people it is meant for. The attacker does not need to break in. They only need to make the system do work.
It works when sending a request costs the client less than handling it costs the server. If every request cost the attacker more than it cost you, nobody would bother building a botnet, because the attacker would run out of money first.
Ordinary web traffic keeps that ratio survivable. A request costs the client a few hundred bytes of bandwidth. It costs the server a database lookup and a few milliseconds of CPU, a tiny fraction of a cent, so it takes enormous volume before the server side hurts.
A model endpoint breaks the ratio. The same few hundred bytes of JSON now buys seconds of GPU time and a response billed by the token. One ordinary response from a frontier model costs a few cents, and one long agent turn carrying a large context can cost dollars, which is several orders of magnitude more per request than anything else in a typical stack. Put a model behind a URL and the amount of traffic you can absorb before it hurts drops by the same factor.
The loop has a valid token
The more likely version of this has no attacker in it.
An engineer points their coding agent at the gateway endpoint. The agent retries on failure, the retry produces the same failure, and there is no budget in the loop to stop it. Left running, one seat can work through a meaningful share of the month's budget before anyone looks at a dashboard. The requests were authenticated, authorized, well-formed, and correctly routed. Every check passed, because every check was designed to catch a stranger.
Traditional rate limiting assumes a person typing, or a service with a call pattern someone reviewed. An agent is neither. It is a program that emits requests at whatever rate its control loop allows, with your credentials, on behalf of somebody who has done nothing wrong and has no idea it is happening.
Design for that first. The malicious case is a subset of it, and the easier subset.
Rate limit the identity, not the address
Per-IP limits are the reflex, and they fail here in both directions.
Twenty engineers behind one office egress or one VPN pool share an address, so a limit generous enough for the team is generous enough for any one member to exhaust it. Meanwhile a single engineer runs several agents at once from that same address, and you cannot tell them apart. The address stopped identifying anything useful.
Scope limits to identity instead. The immediate payoff is that budgets become a policy whoever owns the AI bill can actually set: a larger allowance for the platform team, a smaller one for the department that is experimenting. Enforcing that on addresses is impossible. Enforcing it on identity is a lookup, and the edge products already allow it: Cloud Armor, for one, will key a limit on a header or a cookie rather than on the address.
If your agent runtime issues a credential per run, that credential is the unit worth budgeting against, and it gives you something better than a per-user limit. You get a per-run one, so a single misbehaving loop hits a ceiling without taking its owner's whole day down with it.
Keep the limiter away from the router
Where the check runs matters as much as what it checks.
The gate has to be far cheaper than the work it protects, because that is the only thing restoring the cost ratio in your favor. A token bucket lookup at the edge takes microseconds. Put that same decision inside the router and you have moved it behind the expensive part of the system, so a flood now pays for admission with router capacity. The mitigation becomes the amplifier.
This matters more than it used to, because the router is no longer a cheap component. An intent-aware router like vLLM Semantic Router sits in the request path as an Envoy external processor and runs a classifier on every request to decide where it goes, plus prompt inspection if you are making routing decisions on content. That is cheap next to a frontier model and expensive next to a counter in Redis. It is the wrong place to find out you should have said no.
So the trust zones fall out cleanly.
One VPC in one region: a public subnet, a private subnet, and a managed database beside the cluster
Engineers and their agents
- a coding agent
- a scheduled job
- a person, sometimes
One office address. Many identities behind it, and more agents than people.
Public subnet: the DMZ
The gate, and nothing else. Three cheap checks, in this order.
IAP or VPN
only the network gets this far
auth
who is asking, not where from
rate and budget
per identity, per run, cheap
Never placed here
Private subnet
No route from the internet
Kubernetes cluster, stateless
model router
a classifier on every request
model backends
in-cluster GPUs, managed endpoints
ledger
managed database, outside the cluster
The check has to be far cheaper than the work it protects, which is why it sits in front of the router rather than inside it. Everything that costs money to run lives one subnet back, reachable from the gate and from nowhere else.
Nothing sits in the public subnet (the DMZ, the only part reachable from outside) except the gate. The router, the backends, and anything holding data live in the private subnet, reachable only from the gate's IP range. The ledger stays outside the cluster, so losing the cluster does not lose the record of what it spent, and so the cluster itself stays stateless and easy to replace.
Limits are configuration, not code
The moment you want to change a limit is during an incident, the same moment it is risky to ship a deployment.
If the limits live in router config, every adjustment rolls the data path. Onboarding a contractor becomes a deploy. Cutting a runaway seat at 2am becomes a deploy of the component currently on fire. That is backwards.
Keep the numbers in something the on-call can change without touching the thing under load: Terraform for the durable policy, and a control plane the gate reads for the emergency lever. The limit you most need to work is the one you are writing in the middle of an incident, while spend is still climbing.
Count money, not requests
Requests per minute is the wrong unit for a model endpoint. Two requests that look identical at the edge can differ by three orders of magnitude in what they cost you, so a request budget constrains the wrong variable and gives you false confidence while doing it.
Budget the thing you are actually spending. Tokens per hour and dollars per day, per identity, with a circuit breaker on cumulative spend rather than arrival rate. A limit denominated in money is the only kind you can explain to the person paying the bill.
Choosing what to count
| Requests per address | Tokens and money per identity | |
|---|---|---|
| Twenty engineers behind one egress | One limit shared by all of them | One budget each |
| Several agents from one seat | Indistinguishable | Separate, and per run if the runtime issues a credential per run |
| Two requests a thousand times apart in cost | Counted the same | Counted by what they cost |
| A retry loop overnight | Trips only if it is fast | Trips when it gets expensive |
| Explaining it to whoever pays | A rate they cannot picture | A number on the bill |
Then attribute every dollar to an identity in the ledger, so that when spend spikes you know who to call. The cause is almost always a retry setting in one engineer's agent configuration, and you cannot tell them that unless the trace says whose traffic it was.
What we take from it
A routing gateway is sold on spend control. Built without identity-scoped budgets, it is a spend amplifier with better dashboards, because you have made it easier to send traffic and given yourself no way to stop any of it.
Design the budget model before the routing table. The routing decisions are the interesting engineering, and they are worth doing well, but they only ever save a percentage. The missing budget is what produces the invoice nobody can explain.
If you are standing up a gateway to get token spend under control, we are happy to walk through the budget and trust-zone design before you commit to a topology. Let's chat.
Sources
- vLLM: Semantic Router, the next phase in LLM inference
- Wang et al.: When to Reason, Semantic Router for vLLM
- Envoy: the external processing filter
- Google Cloud Armor: rate limiting overview, and the keys it can enforce on
- Google Cloud: Identity-Aware Proxy overview
- Anthropic: model pricing per million tokens
- GMI Cloud: cost per million tokens on GPU cloud, with H100 throughput figures