Skip to content
American owned and operated
Nightshift

Infrastructure

Rate limiting an LLM gateway, when the attacker is your own engineer's agent

A gateway that routes prompts to cheaper models is a cost control. It is also the most expensive open port you will ever run, and the first thing to overload it is usually a coding agent retrying in a loop with a token you issued.

Nightshift EngineeringSeptember 8, 20266 min read

A gateway that routes prompts to cheaper models is a cost control. It is also the most expensive open port you will ever run, and the first thing to overload it is usually an engineer's coding agent, retrying in a loop, holding a token you issued.

That single failure mode changes where the rate limiter goes, what it counts, and who it is scoped to. Most of the defaults are wrong. This piece is about the shape of the problem, the placement that follows from it, and what to count once it is in place.

Why denial of service works at all

A denial of service attack is a flood of requests aimed at a system until it can no longer serve the people it is meant for. The attacker does not need to break in. They only need to make the system do work.

It works when sending a request costs the client less than handling it costs the server. If every request cost the attacker more than it cost you, nobody would bother building a botnet, because the attacker would run out of money first.

Ordinary web traffic keeps that ratio survivable. A request costs the client a few hundred bytes of bandwidth. It costs the server a database lookup and a few milliseconds of CPU, a tiny fraction of a cent, so it takes enormous volume before the server side hurts.

A model endpoint breaks the ratio. The same few hundred bytes of JSON now buys seconds of GPU time and a response billed by the token. One ordinary response from a frontier model costs a few cents, and one long agent turn carrying a large context can cost dollars, which is several orders of magnitude more per request than anything else in a typical stack. Put a model behind a URL and the amount of traffic you can absorb before it hurts drops by the same factor.

The loop has a valid token

The more likely version of this has no attacker in it.

An engineer points their coding agent at the gateway endpoint. The agent retries on failure, the retry produces the same failure, and there is no budget in the loop to stop it. Left running, one seat can work through a meaningful share of the month's budget before anyone looks at a dashboard. The requests were authenticated, authorized, well-formed, and correctly routed. Every check passed, because every check was designed to catch a stranger.

Traditional rate limiting assumes a person typing, or a service with a call pattern someone reviewed. An agent is neither. It is a program that emits requests at whatever rate its control loop allows, with your credentials, on behalf of somebody who has done nothing wrong and has no idea it is happening.

Design for that first. The malicious case is a subset of it, and the easier subset.

Rate limit the identity, not the address

Per-IP limits are the reflex, and they fail here in both directions.

Twenty engineers behind one office egress or one VPN pool share an address, so a limit generous enough for the team is generous enough for any one member to exhaust it. Meanwhile a single engineer runs several agents at once from that same address, and you cannot tell them apart. The address stopped identifying anything useful.

Scope limits to identity instead. The immediate payoff is that budgets become a policy whoever owns the AI bill can actually set: a larger allowance for the platform team, a smaller one for the department that is experimenting. Enforcing that on addresses is impossible. Enforcing it on identity is a lookup, and the edge products already allow it: Cloud Armor, for one, will key a limit on a header or a cookie rather than on the address.

If your agent runtime issues a credential per run, that credential is the unit worth budgeting against, and it gives you something better than a per-user limit. You get a per-run one, so a single misbehaving loop hits a ceiling without taking its owner's whole day down with it.

Keep the limiter away from the router

Where the check runs matters as much as what it checks.

The gate has to be far cheaper than the work it protects, because that is the only thing restoring the cost ratio in your favor. A token bucket lookup at the edge takes microseconds. Put that same decision inside the router and you have moved it behind the expensive part of the system, so a flood now pays for admission with router capacity. The mitigation becomes the amplifier.

This matters more than it used to, because the router is no longer a cheap component. An intent-aware router like vLLM Semantic Router sits in the request path as an Envoy external processor and runs a classifier on every request to decide where it goes, plus prompt inspection if you are making routing decisions on content. That is cheap next to a frontier model and expensive next to a counter in Redis. It is the wrong place to find out you should have said no.

So the trust zones fall out cleanly.

Figure 1: the gate is the only thing the outside can reach

One VPC in one region: a public subnet, a private subnet, and a managed database beside the cluster

Engineers and their agents

  • a coding agent
  • a scheduled job
  • a person, sometimes

One office address. Many identities behind it, and more agents than people.

bearer token

Public subnet: the DMZ

The gate, and nothing else. Three cheap checks, in this order.

IAP or VPN

only the network gets this far

auth

who is asking, not where from

rate and budget

per identity, per run, cheap

Never placed here

model routermodel backendsledgerprompts
from the gate's range only

Private subnet

No route from the internet

Kubernetes cluster, stateless

model router

a classifier on every request

model backends

in-cluster GPUs, managed endpoints

every dollar, attributed

ledger

managed database, outside the cluster

The check has to be far cheaper than the work it protects, which is why it sits in front of the router rather than inside it. Everything that costs money to run lives one subnet back, reachable from the gate and from nowhere else.

Nothing sits in the public subnet (the DMZ, the only part reachable from outside) except the gate. The router, the backends, and anything holding data live in the private subnet, reachable only from the gate's IP range. The ledger stays outside the cluster, so losing the cluster does not lose the record of what it spent, and so the cluster itself stays stateless and easy to replace.

Limits are configuration, not code

The moment you want to change a limit is during an incident, the same moment it is risky to ship a deployment.

If the limits live in router config, every adjustment rolls the data path. Onboarding a contractor becomes a deploy. Cutting a runaway seat at 2am becomes a deploy of the component currently on fire. That is backwards.

Keep the numbers in something the on-call can change without touching the thing under load: Terraform for the durable policy, and a control plane the gate reads for the emergency lever. The limit you most need to work is the one you are writing in the middle of an incident, while spend is still climbing.

Count money, not requests

Requests per minute is the wrong unit for a model endpoint. Two requests that look identical at the edge can differ by three orders of magnitude in what they cost you, so a request budget constrains the wrong variable and gives you false confidence while doing it.

Budget the thing you are actually spending. Tokens per hour and dollars per day, per identity, with a circuit breaker on cumulative spend rather than arrival rate. A limit denominated in money is the only kind you can explain to the person paying the bill.

Choosing what to count

Requests per addressTokens and money per identity
Twenty engineers behind one egressOne limit shared by all of themOne budget each
Several agents from one seatIndistinguishableSeparate, and per run if the runtime issues a credential per run
Two requests a thousand times apart in costCounted the sameCounted by what they cost
A retry loop overnightTrips only if it is fastTrips when it gets expensive
Explaining it to whoever paysA rate they cannot pictureA number on the bill

Then attribute every dollar to an identity in the ledger, so that when spend spikes you know who to call. The cause is almost always a retry setting in one engineer's agent configuration, and you cannot tell them that unless the trace says whose traffic it was.

What we take from it

A routing gateway is sold on spend control. Built without identity-scoped budgets, it is a spend amplifier with better dashboards, because you have made it easier to send traffic and given yourself no way to stop any of it.

Design the budget model before the routing table. The routing decisions are the interesting engineering, and they are worth doing well, but they only ever save a percentage. The missing budget is what produces the invoice nobody can explain.

If you are standing up a gateway to get token spend under control, we are happy to walk through the budget and trust-zone design before you commit to a topology. Let's chat.

Ready to start building your AI workforce?

It only takes an hour, no fluff. Meet with a Nightshift engineer to get started.