After pretraining on 10B tokens, GPT-2 124M knows the capital of France: given "The capital of France is", it writes "Paris". But ask it a question and it keeps writing more questions:
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of France?
A: The capital of France is Paris.
Q: …
Supervised fine-tuning (SFT) teaches the missing behaviour. We showed the model 138,268 examples in exactly that shape, a question then an answer then an end-of-text token, from two public datasets: Alpaca-cleaned (52k instructions with longer answers) and Natural Questions (87k real search questions with short answers). Only the answer tokens count toward the loss: the model learns to answer, not to write questions. One pass, learning rate 3e-5, about 25 minutes on one rented L4 GPU.
| A | B | |
|---|---|---|
| val loss on answers | 2.453 | 1.975 |
| 20 factual questions right (greedy) | 7 | 11 |
| "What is the capital of France?" | …then loops | "The capital of France is Paris." (stops) |
(A = the base model before fine-tuning, B = after.)
The loss falls fast for the first few hundred steps, which is the model learning the format, then flattens: what's left is knowledge, and fine-tuning adds very little of that. The answers show both sides:
- Right and brief: "Jupiter." · "William Shakespeare." · "Yes, water is a liquid at room temperature." · "Mount Everest."
- Fluent and wrong: "A spider has a total of 6 legs." · "What gas do plants absorb from the air?" "Gasoline." · "The square root of 81 is approximately 2.14."