Same data, same steps, two sizes. A is a 16M-parameter model (4 layers, 256 wide); B is GPT-2 small's shape (12 layers, 768 wide, 124M parameters). Both train 3,000 steps of 32 × 256 tokens of FineWeb-Edu.
| A | B | |
|---|---|---|
| val loss at step 200 | 6.63 | 6.61 |
| val loss at step 3000 | 5.282 | 5.055 |
| tokens per second (L4) | 31.8k | 16.9k |
At step 200 they're tied: early on, both are learning the easy things (which tokens are common). Then the gap opens and keeps widening, to 0.23 nats by step 3,000. The bigger model gets more out of every token it reads. It also costs about twice as much time per step, so per hour of GPU it's closer; scaling-law papers make exactly that trade-off precise (how big a model to train for a given compute budget).
The capstone: GPT-2 on 10 billion tokens
The full recipe (Karpathy's build-nanogpt): the 124M model, 1,024-token context, 524,288 tokens per step, 19,073 steps, about 10B tokens. Twice:
The rented run reached a validation loss of 3.055, GPT-2 small's level, and learned that the capital of France is Paris on the way. It ran at 3.48M tokens/s, about 70× the home GPU, after one of its first machine's eight GPUs turned out to be faulty and slowed all eight (data parallel waits for the slowest GPU).