› Tokenizer Engine

A model reads tokens, not letters

A computer stores text as bytes: one number from 0 to 255 for each plain letter, and two or more for letters like ó or ñ. A tokenizer starts with those 256 bytes, so a byte's number is its ID. Then it learns from a pile of text. Each merge finds the pair of neighbors that shows up most and glues it into one new token, so every merge makes the text shorter in tokens. It stops when no pair shows up twice.

A tokenizer is cheapest for the text it learned from. After a few merges on the English story, the Spanish sentence costs more tokens than the English one. Train on the Spanish story and it flips. Train on both and the gap shrinks. A tokenizer that never saw ó spends two tokens on it, one per byte. Chatbots count and charge by the token too, which is one reason the same question can cost more in some languages.

Your vocab.json and merges.txt are the same two files GPT-2-style tokenizers use, and Hugging Face loads them in Python. Next, see what a model does with tokens in Next Word, a card game where you count which word comes next.

Nothing you make leaves this device. The texts are written for Hi, Bot, and the coins are pretend. More engines →