inference · serving · kv cache
Mooncake: keeping the KV cache so the GPUs don’t redo the work
Chats and agents send the same long text again and again. Recomputing it wastes GPU time; keeping it overflows GPU memory. How big that problem is, how Moonshot AI and Tsinghua University built Mooncake to keep the cache across a whole cluster, what it achieved for Kimi, and what else does the same job.
Before you start
This article assumes you know:
- A language model writes its answer one token (a few characters) at a time.
- Big models are served on many GPUs, in many servers, to many users at once.
Ideas it leans on:
- KV cache
- · the keys and values a model keeps for every earlier token, so it never recomputes them.
- Prefix caching
- · keeping the KV cache of text that requests share (a system prompt, a document, the chat so far) so it isn't recomputed.
- Prefill and decode
- · the two phases of answering: reading the whole prompt at once, then writing the answer one token at a time.
- RDMA
- · remote direct memory access: a network card writing straight into another machine's memory.
The same words, over and over
Ask a chatbot a follow-up question and it doesn’t remember the conversation the way you do. It reads the whole thing again: the system prompt, every earlier message, any document you pasted, then your new question. Agents are more extreme: a coding agent sends its instructions, the files it has open and every step it has taken so far, many times per task. The question changes; most of the text doesn’t.
Before it writes a word, a model turns that input into its KV cache: for every token and every layer, the “keys” and “values” that later tokens look back at. That reading step is prefill, and it’s the expensive, compute-heavy half of answering (the PD disaggregation article explains both halves). If two requests start with the same text, their KV caches for that text are identical. Computing it again is pure waste.
The problem: recompute it, or keep it somewhere
So the obvious move is to keep the KV cache and reuse it when the same beginning (the prefix) comes back. That is prefix caching, and every serving engine does some of it. The catch is where to keep it.
- GPU memory is the fastest place and the smallest. It also has to hold the model’s weights and the caches of every conversation being answered right now. Caches kept “just in case” are the first thing evicted, so a returning conversation often finds its cache gone and is recomputed anyway.
- Each GPU server only sees its own requests. If the follow-up lands on a different server, whatever the first server kept is useless to it.
- Recomputing costs real money and time. The user waits longer for the first word (time to first token, TTFT), and the GPUs spend their most expensive work on text they’ve already read.
The other places to keep a cache, the server’s ordinary memory (DRAM) and its SSDs, are far bigger but sit on the wrong side of the GPU, and on someone else’s server when the request moves. Mooncake is a design for using all of that space, across a whole cluster, fast enough to beat recomputing.
How big the problem is
Reuse is common. In a one-hour sample of Kimi’s traffic, Moonshot AI measured average prompts of 8,590 tokens and answers of 182, about 47 to 1, and found that up to 55% of the KV cache could be reused even with unlimited storage; for a chat-with-papers service it reached 90% (Mooncake paper, arXiv). The FAST ’25 paper’s two Kimi traces and one synthetic workload show the same shape (paper):
| Workload (FAST ’25) | Average prompt | Average answer | Share of KV cache reusable |
|---|---|---|---|
| Conversation (Kimi) | 12,035 tokens | 343 tokens | 40% |
| Tools and agents (Kimi) | 8,596 tokens | 182 tokens | 59% |
| Synthetic (public datasets) | 15,325 tokens | 149 tokens | 66% |
DeepSeek sees the same at scale: over one day, 56.3% of the 608 billion input tokens its service read came from its own on-disk KV cache (DeepSeek’s system, not Mooncake) (DeepSeek).
The caches are big. For Llama 3.1 70B, every token costs about 328 KB (320 KiB) of KV cache (keys and values, 80 layers, 16-bit), so one 128,000-token conversation leaves about 42 GB behind, half an H100’s memory. Models with compressed caches need far less: DeepSeek-V3’s latent attention needs about 70 KB per token, roughly 9 GB for the same conversation (calculator). Contexts keep growing: Llama 4 Scout supports 10 million tokens (Meta), and Anthropic made 1 million tokens generally available (Anthropic).
One server can’t hold enough, and can’t share what it holds. Here is what fits on a single 8-GPU server (a DGX H100), and what a pool across 20 servers adds:
Model
Conversation length
640 GB minus 141 GB of weights; this server only
all 2 TB of a DGX H100; this server only
8 × 3.84 TB; this server only, and slower to read back
about 1 TB spared per server, as in the paper; any prefill server can hit it
Mooncake Store can also keep blocks on SSDs; slower than DRAM, still shared
A server’s own SSDs look big, but they only serve requests that land on that server, and reading from them is slower than from memory. The Mooncake paper did the arithmetic on its traces: the roughly 1 TB of DRAM a server could spare holds about 3 million tokens of Llama-70B-sized cache, which in most workloads didn’t reach half of the best possible hit rate; about 50 million tokens nearly reached it, and that needs the DRAM of at least 20 servers (paper). So the problem is as much about sharing as about size: a cache worth having is bigger than one machine, and every prefill server needs to be able to hit it.
Prices show what reuse is worth. API providers charge far less for input they serve from cache: Claude Fable 5.1 lists $10 per million input tokens and $0.25 for cache hits (Anthropic), Gemini 3.1 Pro Preview $2.00 and $0.20 for prompts up to 200k tokens (Google), DeepSeek V4 Pro (off-peak) $0.66 and $0.022 (DeepSeek). A cache hit costs 10 to 40 times less than reading the text afresh; the pricing table puts these side by side.
Where the problem starts. On one server, an engine’s own prefix caching, with spill-over to CPU memory, covers it. With tens of servers, long shared prefixes and an RDMA network, a shared pool like Mooncake starts to pay. Kimi runs it on thousands of servers.
What Mooncake is
Mooncake is the serving platform behind Kimi, the chatbot from Moonshot AI, designed with Tsinghua University and described in the paper “Mooncake: Trading More Storage for Less Computation” (FAST ’25). Its idea fits in one line: put the KV cache at the centre of the design, and spend cheap memory and network bandwidth to save expensive GPU time.
It builds on prefill–decode disaggregation: separate pools of GPU servers read prompts (prefill) and write answers (decode) (why split them). On top of that, Mooncake adds three pieces:
- Mooncake Store, one KV cache pool built from the spare CPU memory and SSDs of every GPU server in the cluster (docs).
- The Conductor, a scheduler that knows what’s cached where and sends each request where it will finish fastest.
- The Transfer Engine, which moves cache blocks between servers over every network card at once (docs).
How it works
One pool, made of many servers. Mooncake Store splits KV caches into blocks of a fixed number of tokens. Each block is named by a hash of its contents and of everything before it, so the same prefix always has the same names and is stored once, whichever request produced it. Blocks live in the pool, can be copied to several servers (replicas), and the least recently used are evicted when it fills (paper; store docs).
A request in four steps. The Conductor picks a prefill server and a decode server. The prefill server loads the reusable prefix from the pool into GPU memory, computes the rest, and writes the new blocks back to the pool. While it computes, the cache streams layer by layer to the decode server’s memory, so the hand-off overlaps with the work. The decode server loads the cache into GPU memory, and the request joins its next batch (paper).
Scheduling around the cache. For each candidate prefill server, the Conductor estimates the time to first token: time to fetch the reusable cache, plus time waiting in the queue, plus time to compute what isn’t cached. It picks the fastest; if none can meet the latency target, it refuses the request (an HTTP 429, “too many requests”) instead of accepting work it would fail. It also copies blocks that many requests want (hot spots) to more servers (paper). Because Kimi kept running into overload as it grew, the longer arXiv version adds early rejection: deciding before prefill whether the decode side will be able to take the request, so no prefill work is wasted on requests that would be dropped later (arXiv).
Long prompts on several servers. A single very long prompt can be split into chunks processed in a pipeline across several prefill servers, which the paper says needs less network traffic than the usual way of splitting long sequences across GPUs (paper).
Moving the cache fast enough. Reuse only pays if fetching a cached prefix is quicker than recomputing it. The paper works out the floor: for Llama-3-70B-sized models and an 8,000-token prefix, at least 6 GB/s per 8-GPU A800 server and 19 GB/s per 8-GPU H800 server. The Transfer Engine uses RDMA, the network card writing straight into the other server’s memory, including GPU memory where that is the best path (GPUDirect RDMA). It splits each transfer into slices sent over different network cards at once, picks cards by their position in the server, and retries over another card when one fails (docs). Moving 40 GB, roughly one 128k-token conversation, it reached 87 GB/s over four 200 Gb/s cards and 190 GB/s over eight 400 Gb/s cards, about 2.4 and 4.6 times faster than over TCP (paper). It also speaks TCP, NVLink inside a server, NVMe over Fabrics, AWS EFA and other transports (README).
What it achieved
The numbers come from the authors, measured three ways; read each with its conditions.
- In production at Kimi. Mooncake runs on thousands of servers and processes more than 100 billion tokens a day. Compared with Moonshot’s previous systems, which were based on vLLM, it handles 115% more requests on NVIDIA A800 clusters and 107% more on H800 clusters (FAST ’25).
- On replayed real traffic. With a stand-in model shaped like Llama 2 70B on eight-GPU A800 servers, 10 prefill and 10 decode servers against 20 servers running vLLM, under a 30-second TTFT limit and 0.1 seconds between tokens, it served about 75% more requests (arXiv). In the FAST experiments, 16 eight-GPU A800 servers per system serving a model with Llama 3 70B’s shape, it raised the number of requests that met the latency targets by 59% to 498% over vLLM, and the shared pool’s hit rate was up to 2.36 times that of a per-server cache, saving up to 48% of prefill time (FAST ’25).
- In simulation. The headline “up to 525% more throughput” is for simulated long-context workloads (arXiv).
Since then, the Mooncake team, working with the SGLang community, served Kimi K2 on 128 H200 GPUs with prefill and decode split (4 prefill and 12 decode servers), at 224,000 tokens per second of prefill and 288,000 of decode (LMSYS).
Who made it, and who uses it
Mooncake came from Moonshot AI, the company behind Kimi, and Tsinghua University; the paper’s authors are from both (FAST ’25). It won the Best Paper award at FAST 2025, the main storage-systems conference (Tsinghua). The code is open source under the Apache 2.0 licence (licence): the Transfer Engine was released in November 2024 and Mooncake Store in March 2025 (README).
It has since become shared plumbing. vLLM and SGLang use it to move KV caches between prefill and decode servers and to share them through Mooncake Store; LMDeploy added Store support in version 0.18.0; TensorRT-LLM uses the Transfer Engine and merged a Store connector in October 2026, not yet in a release when we checked; LMCache and SGLang’s HiCache can use Mooncake Store as their storage layer; NVIDIA’s NIXL transfer library has a Mooncake backend (README; vLLM connector; vLLM store connector; LMCache; NIXL). It joined the PyTorch Ecosystem in February 2026; the project’s own announcement lists Moonshot AI, Alibaba Cloud, Ant Group, JD.com, Tencent, Meituan, Approaching.AI and the LightSeek Foundation as users (PyTorch). Ant Group reports that cache hits through SGLang HiCache and Mooncake cut time to first token by 84% for DeepSeek-R1 compared with recomputing (LMSYS). Its Transfer Engine is used beyond KV caches too: Moonshot’s separate checkpoint-engine uses it to push updated weights to inference servers during reinforcement learning.
Limits
- It needs a fast network. Below the bandwidth floor above, fetching a cache is slower than recomputing it. The paper finds one fully used 100 Gb/s card per 8-GPU A800 server enough to clear it; faster GPUs need more, and on ordinary data-center networking without RDMA the case weakens.
- Reuse has a ceiling. Even with unlimited storage, 40% to 66% of the cache was reusable in the FAST traces and 55% in the arXiv trace (90% only for a chat-with-papers service). Services without long shared prefixes gain less.
- The evidence is the authors’. The comparisons use a stand-in model (shaped like Llama 2 70B or Llama 3 70B) with replayed traces and mid-2024 vLLM (version 0.5.1) as the baseline; engines have added their own prefix caching and offloading since. Some settings, like the load-balancing threshold, are tuned by hand.
- It’s a toolkit, not a product. Running it means operating a metadata service, a store across servers and RDMA networking; some integrations are marked preview (NIXL plugin). On one server, an engine’s own prefix caching with CPU offload is enough (vLLM KV offloading, SGLang HiCache).
Alternatives
Several of these work with Mooncake rather than against it: they decide what to cache, and Mooncake stores or moves it.
| Option | What it does, compared with Mooncake |
|---|---|
| vLLM prefix caching | Reuse inside one engine; Mooncake plugs into vLLM as a connector when the cache should outlive or leave the server. |
| SGLang HiCache | Extends SGLang’s cache from GPU memory to host memory and shared storage; Mooncake is one of its storage backends, alongside DeepSeek’s 3FS. |
| LMCache | A KV cache layer for vLLM and SGLang that decides what to keep; Mooncake Store can sit underneath it. |
| NVIDIA Dynamo KV Block Manager and NIXL | NVIDIA’s equivalent stack: a cache spanning GPU, host memory and storage, and a transfer library that can use Mooncake as a backend. |
| llm-d | An open Kubernetes stack with cache-aware routing, the job of Mooncake’s Conductor; it has added a Mooncake Store integration and a Mooncake deployment guide (llm-d). |
| Unified Cache Manager (UCM) | Open source, from Huawei’s ModelEngine community, with Ascend support alongside vLLM and SGLang: persists KV caches and adds sparse-attention retrieval; its 3 to 10 times latency reduction is the project’s own figure. |
| Storage vendors’ KV tiers: WEKA Augmented Memory Grid, VAST | Commercial: the cache lives on the vendor’s storage system rather than in pooled server memory (compared here); speed-ups quoted are the vendors’. |
| Prompt caching in APIs (Anthropic and others) | The managed version: if you use a model through an API, the provider runs its own cache and charges less for hits; you control only how you shape prompts. |
In this lab
Ask about this article
Something unclear, or something we got wrong? Use the Ask button at the bottom right; it stays with you as you read. Answers come from this article, sometimes with a small chart, and each shows what it cost. To flag a passage, select it.
Sources
- Mooncake: Trading More Storage for Less Computation (FAST ’25)
- Mooncake: a KVCache-centric disaggregated architecture (arXiv)
- Mooncake on GitHub (README, licence, integrations)
- Mooncake docs: Transfer Engine
- Mooncake docs: Mooncake Store
- Tsinghua University: FAST 2025 Best Paper
- PyTorch: Mooncake joins the PyTorch Ecosystem
- LMSYS: Kimi K2 on 128 H200 GPUs
- LMSYS: SGLang HiCache
- vLLM: Mooncake connector
- vLLM: Mooncake store connector
- vLLM: KV offloading connector
- vLLM: automatic prefix caching
- LMCache: Mooncake backend
- NIXL: Mooncake plugin
- NVIDIA Dynamo: KV Block Manager
- llm-d
- llm-d: Mooncake Store integration
- Unified Cache Manager (UCM)
- WEKA Augmented Memory Grid
- Moonshot AI checkpoint-engine
- DeepSeek-V3/R1 inference system overview
- NVIDIA DGX H100 user guide
- Meta: Llama 4 (10M-token context)
- Anthropic: 1M context generally available
- Anthropic pricing
- Anthropic prompt caching
- Gemini API pricing
- DeepSeek API pricing