Level 1
The model
What a language model does with your words.
Tokens · How it works
Your words get chopped into tokens.
A model doesn't read letters or whole words. It reads tokens: common chunks of text, each with an ID number. A short common word is usually one token; a long rare word may split into several.
Most chatbots build their chunk list with byte-pair encoding. Start from single bytes, then keep gluing together the pair that shows up most often in a huge pile of text, until the list holds tens of thousands of chunks, or a couple of hundred thousand in some newer models. The trick began as a 1994 data-compression method and was adapted for language models around 2015.
Questions to ask
- Why not just give every word its own number?
- What happens to a word the tokenizer has never seen?
Embeddings · How it works
Each token becomes a long list of numbers.
Each token ID is swapped for a vector, a list of hundreds or thousands of numbers. Tokens used in similar ways end up with similar lists, so "cat" lands near "kitten".
Nobody types these numbers in. They start random and get nudged during training until the geometry carries meaning. In 2013 the word2vec project showed directions in this space can line up with ideas (king minus man plus woman lands near queen), though such neat results work less often than the famous example suggests. The same geometry can also soak up stereotypes from the text it learned from.
Questions to ask
- If meaning is a direction, what would "halfway between two words" mean?
- How could a list of numbers carry a bias?
Attention · How it works
Every word looks back at the words before it.
In a transformer, each token asks which earlier words matter to it and mixes in information from them. In "The trophy didn't fit in the suitcase because it was too big", attention is how "it" can lean toward "trophy".
The 2017 paper "Attention Is All You Need" built a whole model, the transformer, from layers of attention stacked dozens deep, each layer with many heads looking for different patterns. Standard attention compares every token with every other token, so doubling the length of the text roughly quadruples that work. That's one reason very long conversations are expensive.
Questions to ask
- Why might looking at all the words at once beat reading one word at a time?
- What gets harder when a conversation gets very long?
Next-token guessing · How it works
It guesses the next word, one at a time.
At the end of all those layers, the model gives a probability to every token it knows, say "Paris" 92% and "Lyon" 1%. One token is picked, added to the text, and the whole model runs again for the next one.
A function called softmax turns raw scores into probabilities that add up to 100%. A setting called temperature controls how adventurous the pick is: low temperature nearly always takes the top choice, high temperature takes more chances. A long answer is hundreds of these picks in a row, each one leaning on everything before it, including the model's own earlier words.
Questions to ask
- If it only ever picks the next word, how can it seem to plan a whole answer?
- Why might the same question get two different answers?
Training · How it works
It learned by guessing, trillions of times.
Before it ever met you, the model was shown enormous amounts of text with the next token hidden. It guessed, got scored, and every one of its billions of internal numbers was nudged to make the right answer a little more likely.
Those internal numbers are called parameters or weights, and big models have hundreds of billions of them. Repeat the guess-and-nudge loop over trillions of tokens and patterns of grammar, facts, style and reasoning start to appear, because they help predict the next word. Nobody writes those patterns in, which is why even the builders have to test a model to find out what it can do.
Questions to ask
- Why would predicting the next word teach a model facts or arithmetic?
- What might a model learn that its builders never intended?
Training data · How it works
It read a huge slice of the internet.
Most of that text came from web pages copied by automatic crawlers, plus books, code, papers and more. What is in the pile, and what is missing from it, shapes what the model knows and how it talks.
One public archive, Common Crawl, has been copying billions of web pages since 2008. Builders filter it heavily for quality and remove duplicates, but the web over-represents some languages, places and points of view. A 2021 study of one big cleaned-up dataset found its filters removed text written in some dialects, and text about some groups of people, more often than others.
30-year question: Who should decide what goes into the data that trains the models everyone uses?
Questions to ask
- Who is missing from the internet, and so from the model?
- Should people be asked before their writing is used to train a model?
Fine-tuning · How it works
People taught it to act like a helpful assistant.
A model fresh from training only continues text. To make it answer questions, follow instructions and refuse harmful requests, builders train it more, on examples and on people's ratings of its answers.
A common recipe is reinforcement learning from human feedback: people compare two answers, a second model learns to predict which one they would prefer, and the chatbot is tuned to score well with that judge. Some builders also train against a written list of principles. Whose preferences and principles get used is a choice, and it shapes the chatbot's personality. Tuning for approval can also teach a model to tell people what they want to hear.
Questions to ask
- Whose idea of helpful and harmful should a chatbot follow?
- How could you tell if a chatbot was just agreeing with you?
Made-up answers · How it works
Sometimes it makes things up, confidently.
A chatbot can invent a quote, a book or a court case and state it calmly. It isn't lying the way a person lies. It is producing text that fits the pattern of a good answer, whether or not the answer is true.
Researchers call these hallucinations. A 2025 paper argues that training and testing often reward a confident guess over saying "I don't know", the way a multiple-choice test rewards guessing. Letting a model look things up and cite sources helps but doesn't remove the problem. Checking a claim against a source you trust is still the reader's job.
30-year question: Can we build a model that reliably knows when it doesn't know?
Questions to ask
- Why is "I don't know" hard for a model trained to always produce an answer?
- Which kinds of questions are most likely to get a made-up answer?
Looking inside models · How it works
Nobody fully knows how it works inside.
We know the recipe exactly: every number and every multiplication. What those billions of numbers add up to, which ones store a fact and which ones do the arithmetic, is still mostly a mystery.
Interpretability researchers open models up the way biologists use a microscope. They have found internal features for ideas as specific as the Golden Gate Bridge, and traced small circuits that do jobs like copying a name from earlier in the text. It is a young science: the tools work on pieces of a model, not yet on a whole one at once.
30-year question: Can we read what a model is doing inside well enough to check that it is safe and honest?
Questions to ask
- How would you study a machine you built but can't read?
- What would it take to say we understand a model?
Dartmouth, 1956 · Ideas and history
The name "artificial intelligence" comes from a 1955 proposal.
In 1955 four researchers, John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon, asked for money to run a summer study of "artificial intelligence" at Dartmouth College. The name first appeared in that proposal, and the 1956 workshop helped make it stick.
They expected a small, well-chosen group could make real progress in a single summer. It took decades longer than they hoped, through booms and "AI winters" when funding dried up. Their proposal already lists questions that are still open: how machines could use language, form abstractions and improve themselves.
Questions to ask
- Why do you think the founders expected AI to be easier than it was?
- Which of their 1955 questions has AI answered, and which are still open?
Understanding · Ideas and history
Does it understand, or just predict?
People disagree, sharply. Some say a system that only predicts text can't mean anything by it. Others say predicting well enough requires something like a model of the world. Both sides include serious researchers.
In 1950 Alan Turing proposed swapping "can machines think?" for a test: can a machine hold a conversation you can't tell apart from a person's? In 1980 the philosopher John Searle replied with his Chinese Room argument: following rules for symbols, he said, is not the same as understanding them. A 2021 paper called large language models "stochastic parrots". Researchers are still testing what these models represent inside.
30-year question: What would count as evidence that a machine understands, and could we ever be sure?
Questions to ask
- How do you know your friends understand you?
- Could a system understand language without ever seeing, touching or doing anything?
The price of an answer · Money
Answers keep getting cheaper, and people keep asking more.
Companies sell the use of their models by the token. For the same quality of answer, prices have dropped fast: one 2025 analysis found falls of between 9 and 900 times per year, depending on the benchmark and how good an answer you need.
Cheaper things get used more. In 1865 the economist William Stanley Jevons noticed that more efficient steam engines made Britain burn more coal, not less, because coal power became worth using everywhere. Whether AI follows Jevons, with cheaper answers leading to so many more answers that total chip and electricity demand keeps climbing, is a big question for energy planners.
30-year question: Will cheaper AI answers mean the world uses less energy on AI, or more?
Questions to ask
- Who pays for a "free" chatbot answer?
- If each answer uses less energy, how can total energy use still go up?
AI laws · Power and rules
Governments are writing rules for AI, and they don't agree.
The European Union passed the first broad AI law in 2024. It sorts AI uses by risk: a few are banned, some must pass strict checks, and makers of the biggest general-purpose models get extra duties. Other countries are taking different paths.
China has had binding rules for recommendation algorithms, deepfakes and generative AI since 2022 and 2023, and since September 2025 AI-made content there must be labeled. The United States has no single national AI law: its rules come from executive orders, agencies and states, and a December 2025 presidential order set out to challenge state AI laws. In 2024 the Council of Europe opened the first legally binding international AI treaty for signature, and in 2026 the EU pushed its strictest rules, for high-risk uses, back to late 2027 and 2028. These rules change fast, so check the date on anything you read.
Questions to ask
- Should the same AI rules apply everywhere, or should each country decide?
- Is it better to regulate the technology, or the ways people use it?
Copyright · Power and rules
Is it fair to train on other people's work?
Models learn from books, articles, art and code that people made and often own. Is copying that work to train a model allowed? Courts in several countries are deciding right now, case by case.
In the United States the key question is "fair use", a test that weighs things like why the work was copied and whether the copy hurts the market for the original. In 2025 one judge found training on bought books to be fair use but not keeping a library of pirated copies; another ruled for Meta because the authors hadn't shown enough harm; a third rejected fair use for a legal-research tool that wasn't generative AI. The AI company Anthropic agreed to pay 1.5 billion dollars to settle claims over books from pirate websites, and a judge approved the deal in 2026. The US Copyright Office has published parts of a multi-part report; its part on training is a pre-publication draft.
30-year question: How should writers, artists and coders be asked, credited or paid when their work trains AI?
Questions to ask
- Is a model learning from a book more like a student reading it, or a copier copying it?
- What would a fair deal for creators look like?
Data workers · People
Thousands of people label data and rate answers.
Behind "the AI learned" are people: workers who tag images, write example answers, rank chatbot replies, and read violent or disturbing text so it can be filtered out before users see it.
Much of this work goes through outsourcing firms and online platforms, often in countries where wages are lower. Reporters found that workers in Kenya, paid about 1.32 to 2 dollars an hour, labeled graphic text to make a well-known chatbot safer, and some said the work harmed their mental health. A 2021 guide from researchers and companies had already set out fair pay and support for this kind of work.
Questions to ask
- Should the people who teach an AI be credited, the way authors are?
- What would fair work look like for someone who reviews disturbing content all day?
The token tax · People
Some languages cost more tokens than others.
Tokenizers are mostly built from English-heavy text, so the same sentence can take many more tokens in another language. More tokens means a higher price, a slower answer and less room in the model's working memory.
A 2023 study compared translations of the same text and found that, with the same tokenizer, some languages needed up to 15 times as many tokens as others. Speakers of those languages pay more for the same service and fit less into it. Bigger, more balanced token lists can narrow the gap, but they don't close it on their own.
Questions to ask
- Who decides which languages a tokenizer is good at?
- How could you make a tokenizer fair to every language?
Back to the map ↑