Cross off the bad move

Tonight

Play a tiny game against a paper robot. When it loses, cross off the move that led there. Play again. It stops choosing the crossed-off move. Nobody handed it the right move. It only learned which tries went badly. When a person later says which of two answers they like better, that pick is a score of the same kind, used on a language model.

Say it in one sentenceYou can learn a better habit from a score, even when nobody shows you the right answer.

Already in the library

• GameMatchbox BrainA paper robot learns a game by crossing off losing moves. Good moves stay.Open →
• Field guideAlignmentPointing a model at what people actually wanted, after it has learned to predict.Open →
• Field guideRLHFUsing human picks as the score that changes what the model tries next.Open →

Textbook names

For ages 13–16 and the grown-up. One sentence each. The picture above is the lesson.

Reward

The score at the end, or along the way.

Policy

The habit: what you do in each situation.

Credit assignment

Deciding which earlier step deserves the blame or the praise.

Preference

A person picking the answer they like better, when there is no single right sentence.

Search at answer time

Trying several drafts and keeping the one that scores best, instead of trusting the first.

The rest of this field

Markov decision processes, the Bellman equations, and policy gradients are the formal version of scoring a situation and then changing the habit. PPO, DPO, and rewards you can check by a program are ways to do that to a language model without letting the habit change so fast that it breaks. Monte Carlo tree search is the same idea as trying several drafts, drawn as a tree.