How we chose our first AI models

Every few weeks a new model launches that's faster, cheaper, or more capable than what we're running, and every few weeks we get asked why we haven't switched yet. Sometimes the answer is that we're already evaluating it. Sometimes the answer is that it fails one of a short list of criteria we don't compromise on, no matter how good the benchmark numbers look. Here's what that list actually is.

Cost has to survive a bad month, not just an average one

A model's per-request price looks fine in a spreadsheet built on average usage. It looks different on the day forty kids are simultaneously iterating on a project, hammering the same tool with retries. We price against a busy-day estimate, not an average-day one, because a tool that gets rationed or shut off mid-session because it got expensive is worse than never having offered it. Cost isn't just "can we afford this" — it's "can we afford this to keep working when the room is full."

Safety means we can name the failure modes, not that failures don't happen

No model is unable to produce a bad output — anyone claiming otherwise about their own product isn't being straight with you. What we actually check is whether a model's failure modes are known, documented, and narrow enough to plan around. A model that occasionally gives a wrong answer to a math question is a known, low-stakes failure mode. A model with a track record of producing content wildly outside what's appropriate for an eight-year-old, with no reliable way to filter it, doesn't clear the bar regardless of how capable it is otherwise.

Capability has to match what the age band actually needs

We don't chase the most powerful model available for its own sake. A model that's extraordinary at long-form reasoning is wasted on a nine-year-old building her first chatbot, and paying for that ceiling doesn't buy her anything she'll use. We match model capability to the task and the age band it's serving — the tools available to an eight-year-old and a sixteen-year-old are genuinely different for this exact reason, not as an arbitrary restriction but because the older kid's projects actually need the extra headroom and the younger kid's don't.

Transparency: we need to be able to explain the tool to a ten-year-old

If a mentor can't explain, in plain language, roughly what a model is doing when it answers a question — that it's predicting likely next words based on patterns in text it was trained on, not thinking the way a person thinks — we don't put it in front of kids. A tool nobody in the room can explain becomes magic, and we're specifically trying to teach kids that these tools are built things with mechanisms, not magic. That requirement rules out some genuinely capable tools whose inner workings are opaque even to us, and we're fine with that trade.

What happens when a model fails one of these later

Models get updated after we've already approved them, sometimes in ways that change their behavior without warning. We re-run our evaluation whenever a provider ships a meaningful update, not just at initial adoption, and we've pulled a tool from rotation before when a change made it fail a check it used to pass. Approval here isn't a one-time gate. It's a standing requirement the tool has to keep clearing.

The honest version of this post

We don't have a perfect, static list of approved models — the landscape moves too fast for that to stay true for more than a season. What we have is this set of questions, asked every time, applied consistently regardless of how impressive a new release looks in the announcement post. That process is slower than just adopting whatever's newest. It's also the only version of "we vetted this" that means anything.