Before comparing anything, you need to know how much two runs differ when nothing differs. Runs A and B are identical (6 layers, 384 wide, 30M parameters, learning rate 1e-3, 2,000 steps of 32 × 256 tokens of FineWeb-Edu) except for the random seed, which changes the initial weights and the order of batches.
| A | B | |
|---|---|---|
| best val loss | 5.330 | 5.328 |
| last train loss (one batch) | 5.250 | 5.423 |
| tokens per second (L4) | 21.7k | 21.5k |
The validation losses end 0.002 apart. That is this setup's noise floor: any later comparison that differs by less than about that much is not a real difference. The training losses differ by 0.17, but that is one batch each; single-batch train loss swings far more than the averaged validation loss.
What a healthy curve looks like: a steep fall over the first few hundred steps (from 10.8, a uniform guess, to about 6.5 as the model learns which tokens are common), then a long, slowly bending decline. No spikes, no plateaus, train and validation close together.