GPUs do 16-bit maths several times faster than 32-bit. Mixed precision (bf16 autocast) runs the big matrix multiplies in bfloat16 while keeping the weights, gradients and optimizer state in fp32. Run A uses bf16, run B pure fp32. Same 30M model, batch 64 × 256 tokens of FineWeb-Edu, 2,000 steps, on an NVIDIA L4.
| A | B | |
|---|---|---|
| val loss at step 400 / 1,200 / 2,000 | 5.942 / 5.176 / 4.897 | 5.941 / 5.178 / 4.898 |
| tokens per second | 43.3k (sharing the GPU) | 32.9k (alone) |
| peak GPU memory | 12.8 GB | 15.3 GB |
The losses agree to three decimals at every evaluation: bf16 lost nothing. It was also faster and smaller, and the speed number understates it: A shared the L4 with other lesson runs while B had it to itself (B ran later, alone, after an out-of-memory crash: pure fp32 at batch 64 needs a 3 GB tensor for the logits alone). On a dedicated GPU the gap is larger; on the home RTX A5000, matrix multiplies measured 40 TFLOPS in bf16 against 12 in fp32.
Why bfloat16 and not fp16: bf16 keeps fp32's exponent range (it just drops mantissa bits), so gradients don't underflow and no loss scaling is needed. The numbers are less precise, but training is noisy anyway: the seed alone moves the result more than bf16 does.