The first test of any training loop: give the model one batch and let it train on nothing else. If the loop is correct (data, loss, backward, optimizer), a 30M-parameter model can memorise 8,192 tokens of web text and drive the loss to zero. If it can't, something is broken, and no amount of real training will fix it.
Run A trained 500 steps on one fixed batch (32 sequences × 256 GPT-2 tokens of FineWeb-Edu). Run B is the same model (6 layers, 384 wide, learning rate 3e-3) training normally on fresh batches.
| A | B | |
|---|---|---|
| train loss at step 500 | 0.000 | 6.06 |
| val loss at step 50 | 8.84 | 7.30 |
| val loss at step 500 | 13.35 | 6.08 |
A passes the test: its training loss reaches zero. The loop works.
Now look at A's validation loss. After a brief dip to 8.84 it climbs to 13.35, well past the 10.83 (ln 50,257) of a model that guesses every token uniformly. A didn't just fail to learn English; it became confidently wrong. Having memorised exactly which token follows which in its 8,192-token world, it puts nearly all its probability on those continuations, and cross-entropy punishes confident mistakes hard.
An earlier version of this lesson ran on Tiny Shakespeare characters (archived in the lab); the shape was the same.