training · overview
AI training at scale, in ten steps
From raw web text to a released model: the ten jobs every training run does, from what size each one starts to matter, and who does it. The map our other articles hang off.
Before you start
This article assumes you know:
- A model learns by adjusting numbers (weights) to predict the next word better.
- GPUs do that arithmetic, many of them at once for big models.
Ideas it leans on:
- Token
- · a chunk of text, often part of a word, that a model reads and writes one at a time.
- Checkpoint
- · a saved copy of the weights and optimizer state, so a run can resume after a failure.
Training is a pipeline, not a program
Training a language model is often described as one thing: point a lot of GPUs at a lot of text. In practice it’s ten jobs in a row, done by different tools, often by different teams. A good model needs all of them; a slow or fragile one stalls the rest.
The same ten steps exist whether you train a 124-million-parameter GPT-2 on one GPU, as our lab does, or a 405-billion-parameter model on 16,384 GPUs, as Meta did for Llama 3 (Llama 3 paper). What changes with scale is which steps get hard. This page names each step, says from what size it needs real attention, and lists who does it. Our other articles go deep on single steps; this is the map they hang off.
Sizes are given on one ladder used across these articles: 1 GPU, 1 server (8 GPUs joined by NVLink), 1 rack (up to 72 GPUs in one NVLink domain such as an NVL72), a cluster (hundreds to thousands of GPUs on a scale-out network) and the frontier (tens of thousands and up).
The ten steps on one page
Pick a scale to see which steps start to matter, and select a step for who does it and where to read more.
At what scale?
Select a step.
open sourcecompanycloud
The ten steps
The names under each step are examples of who does it (open source, companies, clouds), not a ranking; as of 20 Oct 2026.
open sourcecompanycloud
1. Collect and clean
Pretraining text comes mostly from web crawls such as Common Crawl, plus code, books and licensed data. Most of it is thrown away: exact and near duplicates, boilerplate, low-quality pages, unsafe content and personal data. This runs on CPU clusters, not GPUs, and it shapes what the model can learn. Our GPT-2 runs used FineWeb-Edu, an education-filtered slice of Common Crawl. For a fine-tune, this step is a few thousand carefully chosen examples instead.
Needs real attention from: 1 GPU
2. Tokenize and pack
Models read numbers, not letters. A tokenizer splits text into tokens and maps each to an id; the ids are packed into fixed-length sequences and written as shards, so the GPUs never wait on text parsing. It runs once per dataset and is cheap next to training. Our GPT-2 run used GPT-2’s own tokenizer: 10 billion tokens stored as 2-byte ids, about 20 GB.
Needs real attention from: 1 GPU
3. Store and load
The durable copy of the data lives in object storage; a hot copy sits on a parallel file system or local SSDs near the GPUs; a data loader turns files into batches. For text, the read rate is modest: our whole dataset is 20 GB, read once over hours. Storage gets hard with images and video, and with checkpoints (step 7). When it does, you buy throughput vs capacity separately: how many GB/s reach the GPUs, not just how many TB fit. The detour data takes through the CPU, and how to skip it, is in GPUDirect.
Needs real attention from: cluster
4. Choose model and parallelism
Pick the architecture (dense, or mixture-of-experts) and the number of parameters. Then work out how it fits. Training with the Adam optimizer in mixed precision needs about 16 bytes per parameter for weights, gradients and optimizer state, so a 7.5B model needs 120 GB before activations (ZeRO paper): more than one 80 GB GPU. (A saved checkpoint is a little smaller, about 14 bytes per parameter, because the gradients aren’t kept.) From here on the model is split across GPUs with data, tensor, pipeline and expert parallelism, usually several kinds at once.
Needs real attention from: 1 server
5. Build the cluster
Accelerators come in servers of 8, or racks of up to 72 joined by a fast scale-up link (NVLink, AMD Infinity Fabric, Google’s ICI); beyond that, servers talk over InfiniBand or Ethernet with RDMA. A scheduler (Slurm, Kubernetes) hands GPUs to jobs and restarts them. Inside one server the vendor has done the wiring; across servers, the network decides how fast every step can go. How the GPUs and network cards are wired is in GPUDirect: many GPUs.
Needs real attention from: 1 rack
6. Run the training loop
Each step runs a batch forward, measures the error, runs backward to get gradients, averages them across GPUs with an all-reduce, and nudges the weights. Repeat millions of times. The number that matters is MFU, the share of the GPUs’ peak arithmetic you actually use: Meta reports 38–43% for Llama 3 405B on up to 16K H100s (Llama 3 paper). The rest goes to waiting on memory, the network and stragglers.
Needs real attention from: 1 GPU
7. Checkpoint and recover
A checkpoint saves the weights and optimizer state so a crash costs minutes, not the run. At one server, failures are rare. At cluster scale they’re routine: in a 54-day stretch of Llama 3 pretraining, Meta logged 419 unexpected interruptions, about 78% from hardware, and still kept over 90% effective training time (Llama 3 paper). That takes fast checkpoint writes and restores, which is where storage earns its keep. Try a crash on the training board, or read what a crash cost us.
Needs real attention from: cluster
8. Evaluate
Held-out loss tells you training is working; it doesn’t tell you the model is useful. That takes benchmarks and human or model judgement, and a check that the test questions never leaked into the training data, which makes scores look better than they are. Evaluation is a small share of compute at every scale, but it decides when to stop. Our lessons score every checkpoint on held-out questions in the lab.
Needs real attention from: 1 GPU
9. Post-train
A pretrained model continues text; post-training makes it answer. Supervised fine-tuning shows it examples of good answers; preference tuning with RLHF (InstructGPT) or DPO (paper) teaches it which of two answers people prefer; reinforcement learning on checkable tasks such as maths and code rewards correct results. A small fine-tune fits on one GPU; large RL runs generate their practice answers with serving engines such as vLLM inside the training loop. See how we taught our GPT-2 to answer.
Needs real attention from: 1 GPU
10. Release
Package the weights in a safe, fast-loading format such as safetensors, write a model card (what it was trained on, how it was tested, known limits) and a licence, and publish it to a registry. This is where training hands over to serving: AI inference at scale picks up from here.
Needs real attention from: 1 GPU
Where scale changes the answer
Every step exists at every size, but the hard ones move as you climb the ladder:
| Scale | What gets hard | Example |
|---|---|---|
| 1 GPU | Fitting the model and keeping the GPU busy (steps 4, 6) | Our 124M GPT-2 on a home RTX A5000: about two days for 10B tokens |
| 1 server | Splitting the work across 8 GPUs; NVLink handles the traffic (4, 6) | The same run on 8× H100: a few hours |
| 1 rack | Larger models fit inside one NVLink domain of up to 72 GPUs (4, 5) | GB200 NVL72 |
| cluster | The network, data loading and checkpoints (3, 5, 7) | GPUDirect: at what scale it matters |
| frontier | Failures every few hours, and keeping utilisation up through them (6, 7) | Llama 3: 15.6T tokens on up to 16K H100s, 419 unexpected interruptions in 54 days (paper) |
One thing doesn’t change: most of the quality is decided before the GPUs start (step 1) and after they stop (steps 8 and 9). The middle is where most of the compute goes.
Ask about this article
Something unclear, or something we got wrong? Use the Ask button at the bottom right. To flag a passage, select it.