inference · serving · overview
AI inference at scale, in ten steps
From a released model to an answer on someone's screen: the ten jobs every serving fleet does, from what size each one starts to matter, and who does it.
Before you start
This article assumes you know:
- A trained model answers by predicting one token after another.
- It runs on GPUs, which have a limited amount of fast memory.
Ideas it leans on:
- KV cache
- · the keys and values a model keeps for every earlier token, so it never recomputes them.
- TTFT and TPOT
- · time to first token, and time per output token after it: the two delays a user feels.
Serving is a fleet, not a model
Once a model is trained (AI training at scale ends with a release), it has to answer people: quickly, for many of them at once, at a price that works. That is inference, and at scale it’s less about the model than about the machinery around it: how the weights are prepared and loaded, how requests share GPUs, where they are sent, and what is reused.
The same ten steps exist for a chatbot on one GPU, like the Playground in our lab, and for a public service on thousands of GPUs. Sizes use the ladder from the training map: 1 GPU, 1 server (8 GPUs), 1 rack (up to 72 GPUs in an NVL72), a cluster (hundreds to thousands) and the frontier.
The ten steps on one page
Pick a scale to see which steps start to matter, and select a step for who does it and where to read more.
At what scale?
Select a step.
open sourcecompanycloud
The ten steps
The names under each step are examples of who does it (open source, companies, clouds), not a ranking; as of 20 Oct 2026.
open sourcecompanycloud
1. Prepare the model
First, make it fit. The weights must sit in the GPU’s memory with room left for the KV cache. A 70B model in 16-bit numbers is 140 GB, more than one 80 GB H100 holds. Quantization to 8 or 4 bits halves or quarters that, at some cost in accuracy that has to be measured, not assumed. This step decides whether a model runs on one GPU, one server or a rack.
Needs real attention from: 1 GPU
2. Load the weights
Before the first answer, tens of GB to about a terabyte have to travel from storage into GPU memory. On one long-running server that happens once. In a fleet that scales up and down, every new replica pays it, and a slow load means users wait. Loaders such as Model Streamer read many parts of a weights file at once, and fast storage paths avoid the detour through the CPU (GPUDirect).
Needs real attention from: cluster
3. Run a serving engine
The engine does two very different jobs: prefill reads the whole prompt in one heavy pass, then decode writes the answer one token at a time. The engines listed below are all open source, and they compete on how well they keep the GPU busy during both. The two phases, and why they behave so differently, are in prefill–decode disaggregation.
Needs real attention from: 1 GPU
4. Batch and manage the KV cache
One user can’t keep a GPU busy; hundreds can. Continuous batching lets requests join and leave a batch at every step, and paging the KV cache in small blocks stops memory being wasted. The vLLM paper reports 2–4× the throughput of earlier systems at the same latency (vLLM paper), the authors’ own measurement.
Needs real attention from: 1 GPU
5. Split across GPUs
Big models are spread over several GPUs inside a server (tensor or expert parallelism). At larger scale, many operators also run prefill and decode on separate pools of GPUs and ship the KV cache between them, which is the subject of prefill–decode disaggregation.
Needs real attention from: 1 server
6. Route and balance
With many copies of a model running, something has to choose where each request goes. The simple answer is the least busy replica. The better one is the replica that already holds this conversation’s KV cache, because it can skip most of the prefill. Routers that know where caches live are a newer layer, built into stacks such as NVIDIA Dynamo and llm-d.
Needs real attention from: cluster
7. Reuse with caching
Many requests start the same way: the same system prompt, the same document, the same conversation so far. Keeping those KV caches, in GPU memory, CPU memory or SSDs, turns repeated prefill into a lookup; SGLang, for example, shares cached prefixes between requests automatically (SGLang paper). At DeepSeek, 56.3% of input tokens over one day were served from an on-disk KV cache (DeepSeek), and API providers price cached input far below fresh input (what prices reveal).
Needs real attention from: 1 GPU
8. Autoscale
Traffic rises and falls through the day; GPUs cost the same either way. Autoscalers add replicas when queues grow and remove them when they shrink. The catch is step 2: a replica that takes minutes to load can’t absorb a spike that lasts seconds, so fast loading and a few warm spares matter as much as the scaling rules.
Needs real attention from: cluster
9. Measure latency and cost
Users feel two delays, TTFT and TPOT; operators count goodput, the requests per second that met both targets, and the cost per million tokens. Watching averages hides the problem: the slowest few percent of requests are what users notice. Public benchmarks such as MLPerf Inference compare hardware and software under fixed rules.
Needs real attention from: 1 GPU
10. Guard and govern
Anything public needs checks on what goes in and what comes out, limits on how much one user can spend, and logs to investigate abuse. Classifier models do much of this cheaply before an expensive model runs. The Ask button on these pages works that way: a small classifier first decides whether a question is about AI at all, and only then does a larger model answer.
Needs real attention from: 1 GPU
Where scale changes the answer
Every step exists at every size, but the hard ones move as you climb the ladder:
| Scale | What gets hard | Example |
|---|---|---|
| 1 GPU | Fitting weights and KV cache; batching enough users (steps 1, 4) | Our Playground: a 124M model, a few users |
| 1 server | Splitting a big model across 8 GPUs (5) | A 70B model in 16-bit needs at least 2 of 8 H100s just for its weights |
| 1 rack | The largest mixture-of-experts models across one NVLink domain (5) | GB200 NVL72 |
| cluster | Separate prefill and decode pools, routing, shared caches, cold starts (2, 5–8) | PD: who runs it |
| frontier | Everything at once, and cost per token as the deciding number (7, 9) | DeepSeek: up to 278 servers (2,224 GPUs) for V3 and R1 in one day (DeepSeek) |
The pattern is the opposite of training. A training run is one huge job that must not stop; a serving fleet is millions of small jobs that must each finish fast. That’s why inference spends its engineering on batching, routing and caching rather than on checkpoints.
Ask about this article
Something unclear, or something we got wrong? Use the Ask button at the bottom right. To flag a passage, select it.