The three numbers that decide what your inference costs

Teams renting GPUs almost always know their monthly bill and almost never know their cost per thousand requests. Those are different metrics, and only the second one tells you whether things are getting better.

The bill goes up when you grow. Cost per thousand requests going up while you grow means the platform is getting less efficient, and that is the signal worth alerting on.

Three numbers set it.

1. Utilisation, measured honestly#

Not "the GPU is allocated". Not nvidia-smi reporting 100 percent, which reports whether a kernel is resident, not whether the device is doing useful work.

The number to track is achieved occupancy against the memory bandwidth ceiling, or in practical terms: tokens per second per GPU against what that GPU can do for that model at that batch size. Most teams that have never measured it are between 10 and 30 percent.

The gap is nearly always the same thing. Requests arrive one at a time and are served one at a time, so the device spends its life waiting for the next one.

2. Batch size, which is really a latency budget#

Batching is the single largest lever, and it is not free: a request that waits to be batched has waited.

Continuous batching in vLLM or TensorRT-LLM changes the shape of the trade considerably, because a new request joins the in-flight batch at the next decode step instead of waiting for the whole batch to finish. Throughput rises several times over with a tail latency cost measured in tens of milliseconds rather than seconds.

The decision is not "batch or do not batch". It is: what is the p99 you have actually promised, and how much of it is available to spend here? A chat interface where a human is reading the output can give up 200 ms of time-to-first-token and nobody notices. A synchronous call inside a checkout flow cannot give up 20.

Write the number down before tuning anything, because every knob past this point trades against it.

3. What happens at zero#

The largest waste in most inference platforms is not inefficiency during traffic. It is the fleet sitting at idle overnight, at full price, because scale-to-zero was never wired up.

The objection is cold start, and it is a real one: pulling a 30 GB model into GPU memory is not a 5-second operation. The mitigations are unglamorous and they work. Keep the weights on a local NVMe cache rather than object storage. Keep one warm replica for the floor and scale the rest. Separate the queue from the workers so a request that arrives during a scale-up waits in line instead of failing.

For anything with a business-hours traffic shape, this is routinely a third of the bill.

The measurement that ties them together#

One metric, on the same dashboard as latency and error rate:

cost per 1k requests = (GPU-hours x hourly rate) / (requests / 1000)

Broken out by model and by route, because an average across a 7B model and a 70B model is a number about your traffic mix rather than your efficiency.

Then two alerts. One when cost per thousand crosses a threshold, which catches a regression. One when it moves more than 20 percent week over week in either direction, which catches a mix change or a config that silently stopped batching.

Build versus rent, briefly#

The question we are asked most often. The arithmetic is less interesting than people expect.

Owned hardware wins on steady, predictable, high utilisation, and the crossover is usually somewhere around 60 to 70 percent sustained utilisation over an 18-month horizon, once power, cooling, spares and the engineer who replaces failed cards are counted honestly.

Below that, rent. The flexibility is worth more than the discount, and a team that cannot yet keep rented GPUs busy will not keep owned ones busy either. Fix utilisation first; it changes the answer.

A worked example#

A recent engagement: an applied-AI team renting far more GPU than they used, serving one request at a time on a fleet sized for peak, with nothing scaling down.

Continuous batching, a KV cache that survived between requests in a session, and scale-to-zero on the two lowest-traffic models. Throughput went up 3.1 times on the same budget, and cost per thousand requests fell from $1.04 to $0.31.

No new hardware, and no change to the model.