Skip to content
Hi, Bot

AI in Plain WordsUnit 4: Language, pictures and action

› AI-25 · Lesson 25 of 50

Tokens

token

BlueskyX

Language models do not read words the way you do. They read tokens: chunks of text that are often a whole word, sometimes part of a word, a space or a punctuation mark. "Unbelievable" might become "un," "believ" and "able." Each token becomes numbers. This is one reason these systems have sometimes stumbled at counting the letters in a word. They never saw the letters one by one.

Think

How would you count the r's in "strawberry" if you only saw it as two chunks?

Try it

For grown-ups

There are three r's. If "strawberry" arrives as chunks such as "straw" and "berry," the model never sees the letters inside each chunk one by one, so it has to have learned the spelling some other way.