Long runs crash: power cuts, preempted cloud machines, a bug at hour 30. A checkpoint is everything needed to carry on exactly where you were: the weights, the AdamW state (two numbers per weight), the learning-rate schedule position, and the random-number state that decides which batch comes next.
Both runs are the same 30M model and 3,000 steps. A saves a checkpoint every 100 steps, B every 1,000. Mid-run, both were killed with SIGKILL (no chance to save anything) and resumed from their last checkpoint. Dashed lines on the chart are the resumed processes.
| A | B | |
|---|---|---|
| killed after step / resumed from | 1,380 / 1,300 | 1,490 / 1,000 |
| steps done twice | ~80 | ~490 |
| checkpoints written (disk) | 33 (11.7 GB) | 7 (2.3 GB) |
| best val loss | 5.09383 | 5.09383 |
Two things happened.
The crash cost B six times more work. Everything after its last checkpoint was lost and redone. On a 55-hour GPT-2 run that's the difference between losing minutes and losing hours. The price of saving often is disk: A wrote five times as much.
The resume was exact. For the steps both processes ran (8 logged points for A, 49 for B), the resumed process logged the same training loss to six decimals as the one that died. Both runs end at a validation loss of exactly 5.09383, identical to each other despite crashing at different points. That only works because the checkpoint holds the random-number state that decides the batch order: restore just the weights and the optimizer, and the resumed run would see different batches and drift.