The learning rate sets how far each update moves the weights. Run A uses 1e-3; run B uses 1e-2. Same 30M-parameter model, same data (FineWeb-Edu), same 100-step warmup and cosine decay, same seed.
| A | B | |
|---|---|---|
| val loss at step 400 | 6.26 | 6.96 |
| val loss at step 1200 | 5.60 | 6.94 |
| best val loss (step 2000) | 5.300 | 6.705 |
A follows the healthy curve from the baseline lesson. B gets stuck: from step 400 to 1,200 its validation loss barely moves (6.96 → 6.94). The updates are so large that each step undoes the last, and the model hovers at a crude solution instead of descending into a better one. B only starts improving after step 1,200, when the cosine schedule has lowered its learning rate, and it ends 1.4 nats behind A.
A 10× higher learning rate didn't make training faster; it made it fail to progress. The earlier version of this lesson (a 0.8M character model on Shakespeare, archived) found the opposite: there, 1e-2 beat 1e-3. The right learning rate depends on the model and the data. Bigger models and bigger vocabularies generally need smaller ones; GPT-2 124M trains at 6e-4.