train.galvyn.xyz
Learning to train models, in public
Notes from pretraining a small GPT from scratch and fine-tuning it (and other small LLMs) to answer questions, on a home RTX A5000 and rented cloud GPUs. Every lesson compares two runs side by side, with the real loss curves.
Learn · explainersGPUDirect: getting data to the GPU without the CPUAnimated paths from storage and between servers, and at what scale it starts to matter.Latest notes
- lesson 1.1Overfit one batchThe first sanity check of a training loop. Train loss hit 0.000 on one batch; validation loss climbed to 13.35, worse than guessing.
- lesson 1.2How noisy is one run?Two runs that differ only in the random seed ended 0.002 apart. That's the noise floor every other comparison has to beat.
- lesson 1.3Too high doesn't always explodeAt 1e-2 a 30M model stalled for 800 steps and finished 1.4 nats behind 1e-3. On a tiny Shakespeare model the opposite was true.
- lesson 1.4Warmup: cheap insuranceAt learning rate 3e-3, starting at full speed cost 1.18 nats for the whole run. With a 200-step ramp the same rate beat the 1e-3 baseline.
- lesson 1.5Tokens vs bytesPer byte of text, GPT-2 tokens predict better (1.15 vs 1.40), read 4.6× more text per step and process more per second.
- lesson 1.6bf16 costs nothingbf16 and fp32 losses agree to three decimals at every evaluation, while bf16 ran at least 1.3× faster in 17% less memory.
- lesson 1.7What a crash costsBoth runs were SIGKILLed mid-training. Checkpointing every 1,000 steps cost 6× more redone work than every 100; both resumed bit-exactly to the same final loss.
- lesson 1.8Break it three waysA data bug gives a loss of 0.03 that means nothing; no LayerNorm explodes to NaN by step 175; std-1.0 init starts at loss 19.5. Learn the tells.
- lesson 1.9Scale, then the real GPT-2124M beats 16M by a widening margin on the same data. Then the full recipe on 10B tokens: 55 hours at home, 48 minutes on 8× H100, val loss 3.055.
- lesson 2.1Teach it to answer138k question → answer examples taught GPT-2 124M to answer and stop: 7 → 11 of 20 factual questions, and fluent, confident mistakes.
Phase 1. Pretrain a small GPT from scratch
A ~200-line GPT, 3M to 124M parameters, trained on FineWeb-Edu (educational web text) and compared two runs at a time.
- 1.1Overfit one batchTrain loss near zero proves the loop works; val loss rising shows memorising isn't learning
- 1.2Baseline, twiceWhat a healthy loss curve looks like, and how big run-to-run noise is
- 1.3Learning rateThe most important knob: too low is slow, too high is noisy or diverges
- 1.4WarmupEarly steps are fragile: a ramp avoids grad-norm spikes at a high learning rate
- 1.5Tokens vs bytesA token carries several bytes of text, so the same model sees more context and learns faster per step
- 1.6PrecisionMixed precision: faster and smaller with the same loss
- 1.7CheckpointingWhat a crash costs, and what state a checkpoint must hold to resume exactly
- 1.8Break itRecognise failures on sight
- 1.9ScaleBigger models learn more from the same data; then the full GPT-2 recipe, at home vs on 8× H100
Phase 2. Fine-tune a pretrained small LLM
Teach a pretrained model to answer: supervised fine-tuning of our own GPT-2 first, then open 0.5B–3B models with full fine-tuning vs LoRA.
- 2.1Teach it to answerSupervised fine-tuning teaches the format (answer, then stop); the facts come from pretraining
- 2.2Pretrained vs scratchThe value of pretraining
- 2.3LoRA vs fullQuality vs memory vs speed
- 2.4Data mattersWhat fine-tuning changes; forgetting
- 2.5OverfittingWhen to stop; picking the best checkpoint
- 2.6EvalHeld-out loss vs task accuracy vs vibe checks
Notes appear as each step is run. The lab where the experiments run is app.train.galvyn.xyz (anyone can watch).