A language model never sees text, only token ids. Run A reads FineWeb-Edu as GPT-2 tokens (a vocabulary of 50,257 word pieces; on this data one token is 4.61 bytes on average). Run B reads the same text as raw bytes (a vocabulary of 256). Same transformer (6 layers, 384 wide), same 2,000 steps of 32 × 256 positions.
The chart is honest but misleading: B's curve sits far below A's because it measures a different thing. A's loss is per token, B's per byte. To compare, divide A's by 4.61 bytes per token:
| A | B | |
|---|---|---|
| best val loss, as measured | 5.300 per token | 1.400 per byte |
| best val loss per byte | 1.150 | 1.400 |
| text read per step | 8,192 tokens ≈ 37.8 KB | 8,192 bytes = 8.2 KB |
| throughput (L4) | 23.7k tokens/s ≈ 109 KB/s | 70k bytes/s = 70 KB/s |
| parameters | 30.1M | 10.8M |
Per byte of text, the token model predicts better (1.15 vs 1.40), reads 4.6× more text per step, and processes more text per second. Each of its 256 context positions holds about 4.6 bytes, so it also sees 4.6× further back.
The byte model has one real advantage: its vocabulary is tiny, so it has no 19M-parameter embedding table (it's 10.8M parameters instead of 30.1M) and it can spell anything, including words it has never seen. That is why byte-level models are an active research area; with today's architectures, BPE tokens win on cost.