inference · serving · economics
Prefill–decode disaggregation: why big model servers split every request in two
Answering a prompt is two jobs with opposite needs: reading the prompt is heavy maths, writing the answer is mostly waiting on memory. Run both on the same chips and they get in each other's way. How splitting them works, what it costs, who runs it, and when it isn't worth it.
Before you start
This article assumes you know:
- A language model writes its answer one token (a few characters) at a time.
- A GPU runs the model's arithmetic and has its own fast memory.
- Popular models are served on many GPUs to many users at once.
Ideas it leans on:
- KV cache
- · the keys and values a model keeps for every earlier token, so it never recomputes them.
- Batching
- · running many users' tokens through the model in one pass; continuous batching lets requests join and leave mid-flight.
- Memory-bound vs compute-bound
- · whether a GPU is waiting on its memory or on its maths units.
- RDMA
- · remote direct memory access: a network card writing straight into another machine's memory.
How a model answers: prefill, then decode
A language model reads and writes tokens, chunks of text a few characters long. It can only ever do one thing: given the tokens so far, predict the next one. So answering a question happens in two stages.
- Prefill. The model reads your whole prompt in a single pass. All of its tokens are known up front, so they are processed together, in parallel. Out comes the first token of the answer, and a KV cache: for every prompt token and every layer, the “keys” and “values” that later tokens need to look back at it. Without the cache, each new token would mean recomputing the keys and values of every earlier token from scratch.
- Decode. Now the answer is written one token at a time, because each token depends on the one before it. Each step runs the whole model again for a single new token, reading the KV cache and appending to it, until the answer ends.
The KV cache is not small. Our own GPT-2 keeps about 37 KB per token; Llama 3.1 70B keeps about 0.33 MB, so one 8,000-token prompt leaves 2.6 GB of cache in GPU memory (where those numbers come from).
Prefill, decode, batching and the KV cache are steps 3 to 7 of the ten in AI inference at scale, which shows the whole serving stack around them.
What users feel: TTFT and TPOT
The two phases map onto the two delays people notice when an answer streams in:
- Time to first token (TTFT): how long before anything appears. Mostly the prefill, plus any wait in the queue.
- Time per output token (TPOT): how fast the text then streams, on average. That is the decode step. Its close cousin, inter-token latency, is each individual gap between tokens, so a single stall shows up there even when the average looks fine.
Services set targets for both, for example a first token within a second and then a token at least every 50 ms. The DistServe paper uses the measure that matters here, goodput: requests served per second while meeting both targets, rather than raw tokens per second (DistServe, OSDI 2024).
The problem: two jobs on the same GPUs
The simple way to serve a model is to put both phases on the same GPUs and batch requests: run several users’ next tokens through the model in one pass, so each read of the weights serves all of them. Then the phases collide. Picture four users mid-answer, each getting a token every 20 ms, when someone pastes in a 4,000-token document. The figure shows one GPU and five users to keep it readable. Real services run this across hundreds or thousands of GPUs serving thousands of concurrent requests, where a stall hits every user in the batch at once:
A long prompt arrives. The GPU can only do one thing at a time, so the 400 ms prefill runs and everyone already mid-answer waits: their next token comes almost half a second late.
- longest wait between tokens (A–D)
- 420 ms
- E’s first token
- 400 ms
- GPUs
- 1
Illustrative timings: a 20 ms decode step for the batch, a 400 ms prefill; chunked, 8 steps of one chunk plus everyone’s decode tokens (55 ms each); a 30 ms KV handoff. Grey blocks are decode steps, orange is prefill (mixed steps a lighter orange), the palest orange is the transfer; each dot is a token reaching a user. Disaggregation uses two GPUs here against one: a second colocated GPU would also ease the crowding. The gain is that no decode step ever waits on a prefill, and each pool can be sized and configured on its own. In production each “GPU” here is a group of 8 to 32 or more GPUs holding one copy of the model, and each pool has many such groups.
Colocating both phases forces three compromises. They interfere: a long prefill freezes everyone’s stream, or, chunked, slows every step. They share one configuration: the same number of GPUs, the same way of splitting the model, the same batch size, even though the phases want different ones. And you can only scale them together, adding GPUs for both even when only one is short. Chunked prefill (Sarathi-Serve) eases the first problem; it doesn’t touch the other two.
The fix: split them
Prefill–decode (PD) disaggregation runs the two phases on separate pools of accelerators. A router sends a new request to a prefill server; it reads the prompt, produces the first token and the KV cache, and hands the cache to a decode server, which writes the rest of the answer. Each pool can then be built for its own job:
- Its own size. The ratio of prefill to decode capacity follows the traffic: long prompts and short answers need more prefill, chatty answers more decode.
- Its own layout. DeepSeek runs prefill in units of 4 servers (32 GPUs) and decode in units of 18 servers (144 GPUs), with the model split differently in each (DeepSeek’s inference write-up; more below).
- Even its own hardware. Decode mostly waits on memory, not maths (why), so Microsoft Research’s Splitwise runs each phase on hardware suited to it and reports 1.4× the throughput at 20% lower cost (Splitwise, ISCA 2024). Chips are now being built for one phase (see the market).
Two jobs with opposite needs
Every pass through the model does two things: it reads all the weights from the GPU’s memory (HBM, high-bandwidth memory stacked beside the chip), and it does arithmetic with them, about 2 operations per weight per token. Which takes longer depends on how many tokens share that one read of the weights.
- Prefill pushes thousands of tokens through each read of the weights. Plenty of arithmetic per byte: it is compute-bound, limited by the GPU’s maths units.
- Decode pushes one token per user. Batching helps, but with a modest batch the GPU spends most of each step waiting for weights to arrive from memory: it is memory-bound. Batch hundreds of users and the weights stop being the bottleneck, but every user’s KV cache still has to be read on every step, so decode stays limited by memory, now by the caches.
An H100 can do about 989 trillion 16-bit floating-point operations a second (without sparsity) but read only 3.35 TB a second, so it needs roughly 295 tokens per pass before its maths units, rather than its memory, set the pace. The figure shows one GPU’s share of the work: a 70B model is split across several GPUs, and a large service runs hundreds of copies. The ratio, and so the verdict, is the same at any size. Move the slider:
Model
Memory-bound. The maths units are busy 0.3% of the time; the rest is waiting for weights.
One H100 SXM’s share of Llama 3.1 70B in 16-bit weights (a big model is split across several GPUs; the ratio is the same): 989 TFLOPS dense BF16 and 3.35 TB/s (NVIDIA). The two meet at about 295 tokens per pass. Ignores attention and KV-cache reads; with hundreds of users in a batch, reading their KV caches becomes the bottleneck instead, so decode stays memory-bound.
This is not an NVIDIA quirk. Every accelerator, whether an AMD Instinct, a Google TPU or a Huawei Ascend, has far more arithmetic than memory bandwidth, so the same split appears on all of them. The two phases want different things: prefill wants raw compute, decode wants memory bandwidth and lots of memory capacity for KV caches.
The price: moving the KV cache
Splitting adds a step that colocated serving never had: the KV cache must travel from the prefill server to the decode server before decoding can start. Whether that is cheap depends on three things: how big the cache is, how long the prompt is, and how fast the link is.
The size comes from the model’s attention design. For standard attention, every token costs 2 (keys and values) × layers × KV heads × head size × 2 bytes. Models like Llama use grouped-query attention, where several query heads share one key/value head; that is why Llama 3.1 70B has 8 KV heads, not 64. Our GPT-2 comes to 36,864 bytes (about 37 KB) per token and Llama 3.1 70B to 327,680 bytes (about 0.33 MB). DeepSeek-V3 uses multi-head latent attention, which stores one compressed 576-value vector per layer instead: 61 layers × 576 × 2 bytes = 70,272 bytes (about 70 KB) per token, so an 8,000-token prompt leaves about 0.56 GB rather than 2.6 GB. Try other models and links:
Model
Prompt
Link from prefill to decode
KV cache
2.6 GB
328 KB per token × 8,000 tokens. 2 (keys, values) × 80 layers × 8 KV heads × 128 × 2 bytes. Llama 3.1 70B config
Sending it
6.6 ms
for the whole server, over one 400 Gb/s NIC per GPU (8 per server) (DGX H100 guide). Real links reach less than line rate.
The prefill itself, at best
143 ms
2 × 70.6B parameters × 8,000 tokens on 8 H100s at their peak. Real prefill is slower, which only helps the handoff.
handoff ÷ prefill = 0.05 · Cheap: the handoff is a small fraction of the prefill it follows.
Compared per 8-GPU server. Real deployments run whole pools of these: DeepSeek runs prefill in 4-server (32-GPU) units and decode in 18-server (144-GPU) units, each handing caches over its own NICs at the same time.
This is why disaggregation lives on fast interconnects. Within a server, GPUs share NVLink (or AMD’s Infinity Fabric, or Huawei’s Unified Bus in a CloudMatrix). Between servers, the cache goes over RDMA (network transfers that write straight into another machine’s memory), ideally from one GPU’s memory directly into another’s; how that works, and what it needs, is in GPUDirect: getting data to the GPU without the CPU. Libraries exist just for this handoff: NVIDIA’s NIXL, the Mooncake Transfer Engine (how it works), LMCache, AMD’s MORI-IO.
Model design matters too: DeepSeek’s latent attention needs about 70 KB per token against Llama 3.1 70B’s 328 KB, about 4.7× less (both at 16-bit; DeepSeek’s own service may store it in fewer bits), which makes it much cheaper to ship. In production the handoff runs between pools: DeepSeek’s 32-GPU prefill units feed its 144-GPU decode units.
When it helps, and when it doesn’t
Disaggregation is not free throughput. vLLM’s documentation says its own disaggregated prefill does not improve throughput (vLLM docs). What it buys is control: you tune time to first token and time per token separately and keep the tail of token delays tight, so more requests meet their targets. That is how to read the gains in the table below: most are goodput, requests served within latency targets (DistServe, Mooncake, the AMD result), or come bundled with other changes such as cheaper hardware for decode (Splitwise) or large-scale expert parallelism (SGLang, DeepSeek).
| Worth it | Probably not |
|---|---|
| Long prompts (documents, code, retrieval, agents carrying long histories) | Short prompts, where a prefill is a blip next to decode |
| Strict streaming targets, many concurrent users | Offline batch jobs that only care about total throughput |
| Large models over many accelerators (typically tens to thousands of GPUs), with fast links between them | One small model on one GPU, or a slow network that makes the handoff cost more than the prefill |
| Enough traffic to keep two pools busy | Low or spiky traffic, where a split leaves one pool idle; chunked prefill on shared GPUs is simpler |
In this lab
Who runs it, and what they report
Disaggregation has moved from research papers into production, across vendors and chips. The numbers below are each team’s own and measured on different setups, so they are not comparable with each other.
| Who | Hardware | What they report |
|---|---|---|
| DeepSeek (V3 / R1 service) | NVIDIA H800 | Separate prefill and decode deployments. About 73,700 input tokens/s per 8-GPU server in prefill (including tokens served from cache) and 14,800 output tokens/s per server in decode; over one day, 608B input tokens (56.3% served from the KV cache on disk) and 168B output tokens. DeepSeek-V3 is a mixture-of-experts (MoE) model: each token uses only 8 of its 256 “expert” sub-networks, so the experts are spread across many GPUs (expert parallelism). At its daily peak the service used 278 servers, 2,224 H800 GPUs (the H800 is the export version of the H100), costing about $87,072 a day at $2 per GPU-hour. |
| Moonshot AI · Mooncake (Kimi) | NVIDIA GPU clusters | A KV cache-centred design with separate prefill and decode clusters; 75% more requests on replayed Kimi traffic (115% and 107% more in production on A800 and H800 clusters), up to 525% more throughput in simulation. Open source and used by vLLM, SGLang and TensorRT-LLM (explained here). |
| SGLang (LMSYS) | 96 NVIDIA H100 | Open-source PD plus expert parallelism for DeepSeek: 52,300 input and 22,300 output tokens/s per server on 2,000-token inputs. LMSYS estimates about $0.20 per million output tokens on your own hardware (their estimate; no method given). |
| vLLM + AMD (MORI-IO) | 8 AMD Instinct MI300X | PD inside a single server: 2.5× the goodput of colocated serving on the same 8 GPUs (Qwen3-235B, 2,000-token prompts; of 100 requests at 8 per second, 73 met both latency targets against 30). A benchmark, not a production fleet. AMD also documents PD with SGLang on MI300X. |
| Huawei · CloudMatrix-Infer | 384 Ascend 910 NPUs and 192 Kunpeng CPUs on one Unified Bus | Separate prefill, decode and KV cache pools; for DeepSeek-R1 quantised to INT8 (8-bit numbers), 6,688 prefill tokens/s and 1,943 decode tokens/s per NPU with time per output token under 50 ms, falling to 538 tokens/s under a 15 ms target. |
| Google Cloud · llm-d on TPU | TPU v6e | A published walkthrough of disaggregated vLLM serving across TPU hosts on GKE, using llm-d, the open, Kubernetes-native serving stack (a CNCF Sandbox project). |
| NVIDIA Dynamo | Hopper, GB200 NVL72 | Up to 30× more requests served for DeepSeek-R1 on GB200 NVL72 with disaggregated serving (NVIDIA’s projection, at FP4, 4-bit numbers, with long input and output sequences), and more than 2× the throughput for Llama 70B on Hopper (FP8, 3K tokens in, 50 out). Both comparisons also change how the model is split across GPUs, so the gain isn’t disaggregation alone. |
| AWS · Neuron | AWS Trainium (Trn2, Trn3) | Documented disaggregated vLLM serving with separate prefill and decode servers (one of each, or several of each), the KV cache moved by RDMA over AWS’s EFA network. Separately, AWS and Cerebras announced Bedrock serving with prefill on Trainium and decode on Cerebras CS-3 (announced, not yet launched). |
| Research: DistServe, Splitwise (Microsoft) | GPUs | DistServe: 7.4× more requests or 12.6× tighter latency targets. Splitwise: 1.4× throughput at 20% lower cost. |
Hardware is starting to specialise too. NVIDIA announced Rubin CPX, a GPU for the prefill of very long contexts with 128 GB of cheaper GDDR7 memory instead of HBM (The Register), and AWS and Cerebras announced a service on Amazon Bedrock with prefill on AWS Trainium and decode on Cerebras CS-3 systems, described as launching in the coming months (Amazon).
What API prices reveal
Public price lists encode the same asymmetry. Each output token needs its own memory-bound decode step, while input tokens share compute-bound prefill passes, and input that matches a cached prefix skips recomputing its keys and values. Prices per million tokens, as listed on 20 October 2026:
| Model | Input | Cached input | Output | Output ÷ input |
|---|---|---|---|---|
| Claude Fable 5.1 | $10 | $0.25 | $50 | 5× |
| Gemini 3.1 Pro Preview (prompts ≤ 200k) | $2.00 | $0.20 | $12.00 | 6× |
| DeepSeek V4 Pro (off-peak; peak hours cost 2×) | $0.66 | $0.022 | $1.98 | 3× |
Output costs 3–6× input, and cached input 10–40× less than fresh input. Two footnotes: Gemini also charges $4.50 per million tokens per hour to keep a cache stored, and DeepSeek’s peak-hour prices are double its off-peak ones, so its ratios are the same at any hour. Prices also reflect business choices, so read them as a hint, not a measurement.
DeepSeek’s own traffic shows why the split pays at scale: 3.6 input tokens for every output token, and more than half of all input served from cache. A fleet like that needs its prefill and decode capacity sized independently.
Beyond two pools
- The KV cache as its own tier. Mooncake and Huawei’s design pool host memory and SSDs across the cluster into a shared cache, so a repeated prefix (a system prompt, a long document, a conversation so far) is fetched instead of recomputed (how much reuse there is, and how Mooncake pools it).
- Splitting the model itself. Huawei’s xDeepServe also separates attention from the mixture-of-experts layers onto different chips, the same idea one level down.
- More phases. For images and audio, encoding the media is a third stage, and serving stacks now split encode, prefill and decode.
Ask about this article
Something unclear, or something we got wrong? Use the Ask button at the bottom right; it stays with you as you read. Answers come from this article, sometimes with a small chart, and each shows what it cost. To flag a passage, select it.
Sources
- DistServe (OSDI 2024): disaggregating prefill and decoding for goodput
- Splitwise: efficient generative LLM inference using phase splitting (arXiv)
- Splitwise at Microsoft Research (ISCA 2024)
- AWS Neuron: disaggregated inference with vLLM on Trainium
- Amazon: AWS and Cerebras disaggregated inference on Bedrock
- The Register: NVIDIA Rubin CPX
- Mooncake: a KVCache-centric disaggregated architecture (paper)
- Mooncake (GitHub)
- Sarathi-Serve: chunked prefills
- DeepSeek-V3/R1 inference system overview
- LMSYS: SGLang with PD disaggregation and large-scale EP on 96 H100s
- NVIDIA: Introducing Dynamo
- vLLM docs: disaggregated prefilling
- vLLM blog: PD disaggregation on one MI300X node (MORI-IO)
- AMD ROCm blog: disaggregating prefill and decode with SGLang
- Huawei: serving LLMs on CloudMatrix384
- Huawei: xDeepServe on CloudMatrix384
- Google codelab: disaggregated TPU vLLM inference with llm-d on GKE
- llm-d
- NVIDIA H100 specifications
- Model configs: Llama 3.1 8B
- Model configs: Llama 3.1 70B
- Model configs: Qwen3-32B
- Model configs: DeepSeek-V3
- Anthropic pricing
- Gemini API pricing
- DeepSeek API pricing