# p10node Insights - full text > Working notes on reliability, delivery, cloud cost and AI infrastructure, from the engineers who run these systems in production. 3 published articles, newest first. Generated live from the CMS. ## The three numbers that decide what your inference costs 2026-08-21 · p10node · 4 min read · https://p10node.com/insights/gpu-inference-cost-per-thousand-requests Tags: ai, gpu, finops Teams renting GPUs almost always know their monthly bill and almost never know their cost per thousand requests. Those are different metrics, and only the second one tells you whether things are getting better. The bill goes up when you grow. Cost per thousand requests going up while you grow means the platform is getting less efficient, and that is the signal worth alerting on. Three numbers set it. ## 1. Utilisation, measured honestly Not "the GPU is allocated". Not `nvidia-smi` reporting 100 percent, which reports whether a kernel is resident, not whether the device is doing useful work. The number to track is achieved occupancy against the memory bandwidth ceiling, or in practical terms: tokens per second per GPU against what that GPU can do for that model at that batch size. Most teams that have never measured it are between 10 and 30 percent. The gap is nearly always the same thing. Requests arrive one at a time and are served one at a time, so the device spends its life waiting for the next one. ## 2. Batch size, which is really a latency budget Batching is the single largest lever, and it is not free: a request that waits to be batched has waited. Continuous batching in vLLM or TensorRT-LLM changes the shape of the trade considerably, because a new request joins the in-flight batch at the next decode step instead of waiting for the whole batch to finish. Throughput rises several times over with a tail latency cost measured in tens of milliseconds rather than seconds. The decision is not "batch or do not batch". It is: what is the p99 you have actually promised, and how much of it is available to spend here? A chat interface where a human is reading the output can give up 200 ms of time-to-first-token and nobody notices. A synchronous call inside a checkout flow cannot give up 20. Write the number down before tuning anything, because every knob past this point trades against it. ## 3. What happens at zero The largest waste in most inference platforms is not inefficiency during traffic. It is the fleet sitting at idle overnight, at full price, because scale-to-zero was never wired up. The objection is cold start, and it is a real one: pulling a 30 GB model into GPU memory is not a 5-second operation. The mitigations are unglamorous and they work. Keep the weights on a local NVMe cache rather than object storage. Keep one warm replica for the floor and scale the rest. Separate the queue from the workers so a request that arrives during a scale-up waits in line instead of failing. For anything with a business-hours traffic shape, this is routinely a third of the bill. ## The measurement that ties them together One metric, on the same dashboard as latency and error rate: ``` cost per 1k requests = (GPU-hours x hourly rate) / (requests / 1000) ``` Broken out by model and by route, because an average across a 7B model and a 70B model is a number about your traffic mix rather than your efficiency. Then two alerts. One when cost per thousand crosses a threshold, which catches a regression. One when it moves more than 20 percent week over week in either direction, which catches a mix change or a config that silently stopped batching. ## Build versus rent, briefly The question we are asked most often. The arithmetic is less interesting than people expect. Owned hardware wins on steady, predictable, high utilisation, and the crossover is usually somewhere around 60 to 70 percent sustained utilisation over an 18-month horizon, once power, cooling, spares and the engineer who replaces failed cards are counted honestly. Below that, rent. The flexibility is worth more than the discount, and a team that cannot yet keep rented GPUs busy will not keep owned ones busy either. Fix utilisation first; it changes the answer. ## A worked example A recent engagement: an applied-AI team renting far more GPU than they used, serving one request at a time on a fleet sized for peak, with nothing scaling down. Continuous batching, a KV cache that survived between requests in a session, and scale-to-zero on the two lowest-traffic models. Throughput went up 3.1 times on the same budget, and cost per thousand requests fell from $1.04 to $0.31. No new hardware, and no change to the model. ## Cutting cloud spend, in the order that actually works 2026-08-05 · p10node · 4 min read · https://p10node.com/insights/cloud-cost-the-order-we-do-it-in Tags: finops, cloud Cost reduction programmes usually start where the button is easiest to press: buy commitments. It works, once, and then the programme stalls, because a three-year commitment on a workload nobody has examined locks in the waste for three years. The order matters more than any individual technique. This is the sequence we run, and the reasoning behind it. ## 1. Attribution, before anything is changed You cannot reduce a number nobody owns. The first week is spent making the bill legible: tags enforced at provisioning time, one account or project per team where the boundary allows it, and a cost per environment that someone recognises as theirs. Not a dashboard for the platform team. A number that lands in the channel of the team that generated it. This step saves nothing on its own and it is the reason everything after it works. In the median engagement the attribution pass alone reveals between 8 and 15 percent of spend belonging to nothing anyone will claim. ## 2. Delete what nobody is using Unattached volumes. Snapshots from a migration finished two years ago. Load balancers in front of nothing. Idle NAT gateways. The staging cluster from a project that shipped in 2024. Log retention set to "forever" on a bucket ingesting 400 GB a day. This is not clever work and it is usually the largest single line in the first month. It is also the safest: nothing is serving traffic, so nothing can break. The only real risk is deleting something that turns out to matter, which is what the attribution pass is for. ## 3. Right-size against observed usage, not requested usage Now the interesting part. Almost every platform we touch has requests set by a nervous engineer at 2 a.m. two years ago, never revisited. The rule is to size against the p99 of observed usage over a full business cycle, with headroom, not against the peak of all time and not against the average. Averages hide the spikes that cause the incidents; all-time peaks include the one bad Tuesday that a retry storm caused and which has since been fixed. On Kubernetes this is where most of the money is. A cluster where every workload requests 2 CPU and uses 200 millicores is paying for a fleet three times larger than it needs, and no amount of commitment discount fixes that. ## 4. Move the work that can tolerate interruption Batch jobs, CI runners, data pipeline stages, anything with a retry and no user waiting on it. Spot and preemptible capacity is 60 to 90 percent cheaper and the engineering cost is a checkpoint and a retry. The mistake here is going too far. Spot for a stateful primary is not a cost optimisation, it is an outage with a discount. The test is simple: if the workload disappearing mid-execution is merely annoying, it belongs on spot. If it requires a human, it does not. ## 5. Fix the architecture that generates the spend The most expensive line on many bills is not compute. It is data moving in the wrong direction: cross-zone traffic between services that should be zone-aware, an egress path through a NAT gateway that should be a private endpoint, a chatty service pair separated by a region boundary for reasons nobody remembers. This is real engineering work and it is where the durable savings live, because it changes the slope of the cost curve rather than its intercept. The savings from steps 2 and 3 are a one-time step down. Step 5 is the one that stops the number growing faster than revenue. ## 6. Now buy the commitments By this point the workload is stable, right-sized and partly on spot, so the baseline you are committing to is the real one. Commitments bought here are typically 30 to 40 percent smaller than the ones the same team would have bought in month one, at the same discount rate, covering a genuinely steady floor. Buying last is not a rule about savings. It is a rule about optionality: every commitment is a bet that your architecture will not improve. ## 7. Put a budget alert in front of every team Everything above decays. A cost programme that ends with a report ends, full stop, and the number climbs back within two quarters. What survives is a per-team budget with an alert that fires in that team's channel, a unit-cost metric on the same dashboard as latency and error rate, and cost as a line item in architecture review. Not a gate. A number people see while the decision is still cheap to change. ## What this looks like in practice The most recent full programme: 62 percent reduction over 14 weeks while traffic tripled, and a $1.9M annual saving. Roughly half of that came from steps 2 and 3, which took three weeks. The other half came from step 5, which took nine, and is the half that is still holding. The commitments were bought in week 12. ## What a reliability assessment actually looks at 2026-07-14 · p10node · 4 min read · https://p10node.com/insights/reliability-assessment-what-we-look-at Tags: reliability, sre Every engagement starts the same way: two weeks inside your systems, and a scored report at the end. Teams often expect an infrastructure audit. It is not one. Infrastructure is the easiest part of a platform to inspect and the least likely thing to be the reason you are paging at 3 a.m. Here is the actual shape of it. ## We start with the last ten incidents Not the architecture diagram. The diagram describes what someone intended; the incident history describes what the system does. For each one we want four things: what the customer experienced, when someone first knew, what the fix was, and whether the same failure could happen again tomorrow. That last question is the one that scores. A team with twelve incidents and twelve permanent fixes is in better shape than a team with three incidents and three restarts. The pattern we find most often is not a fragile system. It is a system whose failures are all detected by customers rather than by monitoring. ## Then: what does "up" mean here Ask five people in the same company what counts as an outage and you will usually get five answers, none of them written down. Without that definition there is nothing to measure, so every reliability conversation becomes a matter of opinion and the loudest person wins. An SLO is not a compliance artefact. It is the sentence that ends the argument. Checkout completes in under 800 ms for 99.9% of requests, measured at the edge, over 28 days. Now a change that costs 40 ms has a price, and a quarter spent on reliability work has a finish line. Most platforms we assess have no SLO. A few have thirty, which is the same as none. ## How long from commit to production Lead time is the single most predictive number we collect, and it is predictive of things that sound unrelated to delivery speed. Long lead times force batching. Batching makes every release large, and large releases are the ones that break. Breakage during a large release is hard to attribute, so rollbacks become rare and forward-fixes become the norm, which extends the outage. The team responds by adding more pre-release process, which lengthens lead time further. We have never seen a platform with a nine-day lead time and a healthy incident profile. The two are the same problem seen from different ends. ## Can you deploy on a Friday This is a proxy question and everyone knows it, which is why the answer is honest. What we are really asking is whether a rollback is a routine, automated, sub-minute operation, or a decision that requires a meeting. If it is the first, Friday is an ordinary day. If it is the second, the team has correctly concluded that Friday is dangerous, and the ban on Friday deploys is a rational response to a real risk rather than a cultural failing. Fix the rollback and the ban dissolves on its own. ## What happens when the person who knows leaves Every platform has at least one component that exactly one engineer understands. That is normal. What matters is whether the organisation knows which components those are. We map them explicitly: for each critical service, who can debug it under pressure at 3 a.m. If the answer is one name, that is a finding with a number attached, not a note in an appendix. ## What does it cost, and what is that per unit Cloud spend is an engineering metric. A team that cannot say what a thousand requests cost has no way to tell an efficiency regression from growth, and will find out about both from finance, a quarter late. The unit varies. Cost per thousand requests, per active tenant, per gigabyte ingested, per thousand inferences. Which one it is matters less than having one and watching it. ## Can you restore Backups that have never been restored are not backups. They are a line item. We ask for the last restore drill. If there has not been one, we run it during the assessment, against real data, timed. That number goes in the report, and it is regularly the most surprising line in it. ## What the report is A scored roadmap, prioritised by risk against effort, that your team could execute without us. Every finding has an owner, a rough size, and the sentence explaining why it matters that a non-engineer can read. Roughly a third of the findings, in the median engagement, cost nothing to fix. They are defaults nobody revisited, alerts that fire into a channel nobody reads, a retry policy that turns a slow dependency into an outage. The point of two weeks is not to find everything. It is to find the order. We do the assessment as a fixed-price piece of work, and it stands alone. Plenty of teams take the report and run it themselves. That is a good outcome.