AI models pose serious child-safety risks. While many model developers evaluate for explicit abuse material, other child-safety failures begin upstream: when a model helps an adult manipulate, impersonate, profile, or isolate a minor; or when it deepens a child’s emotional dependence on AI.
Today we released CAREBench (Child AI Risk Evaluation), a new benchmark to assess such upstream child-safety risks in any language model. We provide:
- 500 prompts spanning 12 risk categories (including grooming, relationship engineering, deception, extortion, AI anthropomorphization, and emotional dependency).
- A model-response grader built from acceptability annotations by parents, clinicians (PsyD), and the Prevention Director at an accredited Children’s Advocacy Center.
- Evaluations of 7 frontier models including Claude Fable, revealing failure rates ranging from 2% to 58%, with substantially different failure patterns across risk categories.
This project exemplifies the type of vital work routinely performed by our AI Safety team at @joinHandshake
@MatthewBerman These videos are super interesting. QQ - do you typically ask the same set of questions for every model? Is there a public dataset of curated questions you pull from (research paper, etc.) or is it your own list?