Skip to content
Hi, Bot

AI in Plain WordsUnit 4: Language, pictures and action

› AI-26 · Lesson 26 of 50

Attention

attention / transformer

BlueskyX

Words take meaning from other words. "Bank" means one thing near "river" and another near "money." Modern language models handle this with a method called attention. For each word, the model weighs how much each earlier word should shape its meaning. That is how it can link "it" back to the right thing several sentences earlier. The design built around attention is called a transformer. It is the "T" in GPT, short for Generative Pre-trained Transformer.

Think

In "The trophy didn't fit in the case because it was too big," what is "it"?

Try it

For grown-ups

"It" is the trophy: a trophy that is too big won't fit. Change "big" to "small" and "it" becomes the case. That one-word swap is why this sentence is a classic test of whether a model uses context.