Learn

The systems behind training, one topic at a time: what problem it solves, why this solution, what it needs, and animated diagrams of the paths the data takes. Every section has its own link.

• training · overviewAI training at scale, in ten stepsFrom raw web text to a released model: the ten jobs every training run does, from what size each one starts to matter, and who does it. The map our other articles hang off.
  1. Training is a pipeline, not a program
  2. The ten steps on one page
  3. The ten steps
  4. Where scale changes the answer
  5. Ask about this article
  6. Sources
• inference · serving · overviewAI inference at scale, in ten stepsFrom a released model to an answer on someone's screen: the ten jobs every serving fleet does, from what size each one starts to matter, and who does it.
  1. Serving is a fleet, not a model
  2. The ten steps on one page
  3. The ten steps
  4. Where scale changes the answer
  5. Ask about this article
  6. Sources
• networking · storage · multi-nodeGPUDirect: getting data to the GPU without the CPUTraining moves data from storage to GPUs, and between GPUs on different servers. Normally every byte detours through the CPU and host RAM. GPUDirect RDMA and GPUDirect Storage remove the detour. What that costs, what it needs, and at what scale it starts to matter.
  1. The use case: one training step
  2. The problem: the long way round
  3. Two layers: RDMA fixes the network, GPUDirect fixes the host
  4. Local NVMe: the same trick inside one server
  5. GPU to GPU across servers
  6. At what scale does it matter?
  7. When it matters, and when it doesn’t
  8. Many GPUs: real systems
  9. Which storage can do this?
  10. What you need
  11. Ask about this article
  12. Sources
• inference · serving · economicsPrefill–decode disaggregation: why big model servers split every request in twoAnswering a prompt is two jobs with opposite needs: reading the prompt is heavy maths, writing the answer is mostly waiting on memory. Run both on the same chips and they get in each other's way. How splitting them works, what it costs, who runs it, and when it isn't worth it.
  1. How a model answers: prefill, then decode
  2. What users feel: TTFT and TPOT
  3. The problem: two jobs on the same GPUs
  4. The fix: split them
  5. Two jobs with opposite needs
  6. The price: moving the KV cache
  7. When it helps, and when it doesn’t
  8. Who runs it, and what they report
  9. What API prices reveal
  10. Beyond two pools
  11. Ask about this article
  12. Sources
• inference · serving · kv cacheMooncake: keeping the KV cache so the GPUs don’t redo the workChats and agents send the same long text again and again. Recomputing it wastes GPU time; keeping it overflows GPU memory. How big that problem is, how Moonshot AI and Tsinghua University built Mooncake to keep the cache across a whole cluster, what it achieved for Kimi, and what else does the same job.
  1. The same words, over and over
  2. The problem: recompute it, or keep it somewhere
  3. How big the problem is
  4. What Mooncake is
  5. How it works
  6. What it achieved
  7. Who made it, and who uses it
  8. Limits
  9. Alternatives
  10. Ask about this article
  11. Sources
• storage · business · marketVAST, WEKA and NetApp: three ways to feed GPUs, and three businessesThe three storage companies most often shortlisted for AI clusters were built on different bets: a parallel file system, a disaggregated shared-everything design over NFS, and the enterprise's existing file store. What each is good at, where it lags, what the evidence says, and what their businesses look like.
  1. Why storage decides how busy the GPUs are
  2. Three bets
  3. How each is built
  4. How a GPU server reads from each
  5. Side by side
  6. What the independent evidence says
  7. Where each came from, and where it’s going
  8. The businesses, in numbers
  9. Good at, and lags at
  10. Which fits whom
  11. Ask about this article
  12. Sources