Unit 2 · Lesson 3
Your dataset audit
Run a five-question audit on any dataset to find its bias.
You've seen that AI copies its examples, and that the edges reveal the gaps. Now here's the tool that puts you in charge: a dataset audit.
A dataset is just the pile of examples a model learned from. Auditing it means asking a few sharp questions before you trust anything built on it. Five questions. You can run them on anything.
You can't check every answer an AI gives. But you can check the examples it learned from once — and know what to expect.
One idea: five questions that expose a dataset
Run these in order. Any "uh-oh" answer is a warning label.
| # | Question | What a bad answer sounds like |
|---|---|---|
| 1 | Who's in it? | "Mostly one kind of person / place / example." |
| 2 | Who's missing? | "We forgot to include ______ entirely." |
| 3 | Who made the labels? | "One tired person, fast, with their own opinions." |
| 4 | When and where is it from? | "Ten years ago, one country, now used everywhere." |
| 5 | What's it being used for? | "Something way bigger than what it was built for." |
Notice none of these ask "is the AI good?" They ask "is the foundation honest?" A shaky foundation makes even clever code unreliable.
Do the thing
Audit a dataset you actually know. Try one of these:
- The photos on your phone. If an AI learned "a normal day" from your camera roll, what would it think normal looks like? Who's over-photographed? Who's never in frame?
- Your class's book list. If a model learned "what a story is" from only those books, whose stories would it never have read?
Walk the five questions and jot one honest answer each.
Quick check. Almost every real dataset fails at least one question — including the ones behind tools you love. The point isn't to find a perfect dataset (there isn't one). The point is to know the flaw so you can watch for it. An audited-and-flawed dataset beats a trusted-and-mysterious one every time.
Why this matters
This is the move that turns you from a user of AI into a check on it.
- Ask about the examples before you argue about the answer. Most AI mistakes are dataset mistakes wearing a confident voice.
- "Where did this learn?" is a fair question to ask about any tool — a search engine, a recommendation, a grade-predicting app. If nobody can answer it, that's your answer.
You finished Unit 2. You now know that behavior comes from data, that the edges reveal the gaps, and that five questions can expose almost any dataset. Next unit, we flip to the other side of the conversation: how you ask, and why a good prompt is really just clear thinking written down.