At the start of training the weights are random and the gradients are large and noisy. Warmup ramps the learning rate up from zero over the first steps instead of starting at full speed. Both runs here use a high learning rate, 3e-3; A ramps up over 200 steps, B starts at 3e-3 on step one.
| A | B | |
|---|---|---|
| val loss at step 200 | 6.63 | 7.03 |
| val loss at step 1200 | 5.57 | 6.58 |
| best val loss (step 2000) | 5.206 | 6.389 |
B never recovers from its first steps. By step 200 it sits at 7.03, exactly where the 1e-2 run in the learning-rate lesson got stuck, and it stays more than a full nat behind A for the rest of training.
A, with the same peak learning rate but a gentle start, ends at 5.206: better than the 1e-3 baseline (5.300). That's the point of warmup: it lets you use a learning rate that would otherwise be too high, and the higher rate then learns faster for the rest of the run.