A pile of chances

Tonight

Write six words on cards. Roll a die to pick the next one. Some words are allowed to follow, and some are not. A language model is that pile, counted from everything it has read, then asked for one word at a time. Turn the pile flatter and rare words show up. Turn it sharper and the model keeps repeating the obvious word.

Say it in one sentenceThe next word is the most likely next card, unless you let the model be more surprising on purpose.

Already in the library

• GameNext WordTwelve cards and one die. The story is counted so each face matches how many words can come next.Open →
• Field guideTokenThe pieces the pile is actually choosing between.Open →
• Field guideStochasticWhy the same prompt can come back with a different sentence.Open →

Textbook names

For ages 13–16 and the grown-up. One sentence each. The picture above is the lesson.

Conditional probability

The chance of this word, given the words already said.

Entropy

How surprised you expect to be. A coin flip is high surprise. A sentence you can finish is lower.

Cross-entropy

How surprised the model is by the real next word. Training pushes that surprise down.

Temperature

A dial that makes rare words more welcome, or less.

Perplexity

Surprise, averaged, so two models can be compared.

The rest of this field

Keeping only the top few words, or the smallest set of words that covers most of the chance, is the same pile with a cutoff. Kullback–Leibler divergence and mutual information are the bookkeeping for comparing two piles of chances.