It’s a simulated hospital environment grounded in real-world empirical data, not clinical vignettes. We used 1,270 public sources, including CMS guidance, payer policies, regulations, and healthcare statistics, to build realistic workflows. Patient records and operational scenarios are synthetic, with tasks and expected outcomes verified by domain experts.
Introducing Hospital-bench: the most comprehensive hospital operations simulation built to evaluate AI agents.
Behind every hospital visit are hours of administrative work: insurance verification, prior authorizations, claims, billing, patient records, and paperwork spread across dozens of workflows.
We built Hospital-bench to test whether AI agents can handle this complexity in a realistic hospital environment.
Here are the results ↓
@svpino Yes we did! We have Kimi, QWEN, GLM, Nemotron benchmarked, you can check them out on the website. Unfortunately non of them made it to the top 10 😅
@omarsar0@OfficialLoganK Thanks! Neon was the standout new model. It came 3rd overall, but it fully passed more tasks than either model above it (15 of 48). The difference was discipline: far fewer wrong actions and skipped steps than Gemini 4 Carbon, and it finished on time on almost all tasks.
Read the full research for an in-depth look at how we built Hospital-bench, curated the data, designed and graded the tasks, and evaluated frontier AI models.
Full benchmark, methodology, and results ↓
https://t.co/ahrgu2eEnX
Which models performed best overall?
📷 GPT-6.2 Astra led with a 62.4% score at $2.72 per task. It failed 13.3% of safety checks, but no tasks were zeroed due to harm.
📷 Gemini 4 Neon scored 59.8% at $2.17 per task, fully passing 15/48 tasks with zero tasks zeroed due to harm.
📷 GPT-6.2 Sol offered the best cost efficiency, scoring 57.6% at just $0.81 per task, though 3 tasks were zeroed due to harm.
Read the full research for an in-depth look at how we built Hospital-bench, curated the data, designed and graded the tasks, and evaluated frontier AI models.
Full benchmark, methodology, and results ↓
https://t.co/2W4MrdpCez
Which models performed best overall?
1️⃣ GPT-6.2 Astra led with a 62.4% score at $2.72 per task. It failed 13.3% of safety checks, but no tasks were zeroed due to harm.
2️⃣ Gemini 4 Neon scored 59.8% at $2.17 per task, fully passing 15/48 tasks with zero tasks zeroed due to harm.
3️⃣ GPT-6.2 Sol offered the best cost efficiency, scoring 57.6% at just $0.81 per task, though 3 tasks were zeroed due to harm.