Three ways to break training, each next to a healthy run (A) so you can learn what broken looks like before it happens to you. Same 30M model and FineWeb-Edu data throughout.
A loss that's too good: targets equal inputs
The model is supposed to predict the next token. In B a one-line data bug feeds it the current token as the target, so it only has to copy its input.
| A | B | |
|---|---|---|
| val loss at step 200 | 6.57 | 0.14 |
| best val loss | 5.300 | 0.027 |
B's loss looks wonderful and means nothing: the model has learned to copy, not to read. The tell is a loss far below anything plausible for real text. GPT-2-sized models end around 3 on web text, so 0.03 is a bug, not a breakthrough.
No LayerNorm: the explosion
LayerNorm rescales the activations inside every block. B removes it, at a learning rate of 3e-3.
| Step | 40 | 60 | 80 | 100 | 140 | 160 | 175 |
|---|---|---|---|---|---|---|---|
| B train loss | 7.75 | 7.51 | 16.8 | 31.2 | 65.8 | 105.6 | NaN |
| B gradient norm | 0.66 | 1.96 | 179 | 560 | 11,004 | 15,057 | – |
For 60 steps B learns normally. Then, with nothing holding the activations at a stable scale, they grow layer by layer, the gradients grow with them, each update makes it worse, and by step 175 the loss is NaN (the trainer stops there). A, with LayerNorm, trains to 5.256. The gradient norm warned about 100 steps before the NaN.
Bad initialisation: std 1.0 instead of 0.02
GPT-2 initialises weights with a standard deviation of 0.02. B uses 1.0.
| A | B | |
|---|---|---|
| val loss at step 200 | 6.57 | 19.55 |
| largest early gradient norm | 5.9 | 52,271 |
| best val loss | 5.300 | 7.625 |
With weights 50× too large, the first logits are enormous and confidently wrong: B's validation loss at step 200 is 19.55, almost twice the 10.83 of a uniform guess. Gradient clipping (to a norm of 1.0) keeps it from blowing up the way the LayerNorm run did, but it never recovers the lost ground and ends 2.3 nats behind.