AI in Plain WordsUnit 4: Language, pictures and action
› AI-26 · Lesson 26 of 50
Attention
attention / transformer
Words take meaning from other words. "Bank" means one thing near "river" and another near "money." Modern language models handle this with a method called attention. For each word, the model weighs how much each earlier word should shape its meaning. That is how it can link "it" back to the right thing several sentences earlier. The design built around attention is called a transformer. It is the "T" in GPT, short for Generative Pre-trained Transformer.
Think
In "The trophy didn't fit in the case because it was too big," what is "it"?
Try it
For grown-ups
"It" is the trophy: a trophy that is too big won't fit. Change "big" to "small" and "it" becomes the case. That one-word swap is why this sentence is a classic test of whether a model uses context.