AI in Plain WordsUnit 4: Language, pictures and action
› AI-25 · Lesson 25 of 50
Tokens
token
Language models do not read words the way you do. They read tokens: chunks of text that are often a whole word, sometimes part of a word, a space or a punctuation mark. "Unbelievable" might become "un," "believ" and "able." Each token becomes numbers. This is one reason these systems have sometimes stumbled at counting the letters in a word. They never saw the letters one by one.
Think
How would you count the r's in "strawberry" if you only saw it as two chunks?
Try it
For grown-ups
There are three r's. If "strawberry" arrives as chunks such as "straw" and "berry," the model never sees the letters inside each chunk one by one, so it has to have learned the spelling some other way.