CRASH LAB

The Centre for Responsible Autonomous Systems in Healthcare is a clinician-led collaborative research group anchored at KCDH-A

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

⚡️Our 4th metric: RadLE-S = Safety Index.

This measures protection against confident wrong diagnoses and one of the most important ones! A wrong answer with high confidence is very dangerous and RadLE-S penalizes wrong answers more heavily when confidence is high.

🏆 Top AI models on RadLE-S:

1. Claude Fable 5 — 56.8 👑
2. OctoMed 7B — 52.3
3. Meta Muse Spark 1.1 — 48.0

Human expert baseline: 67.0

This is where some models really suffer as they make mistakes confidently.

A confident wrong answer can mislead a trainee, reassure a patient falsely, or push a clinician toward the wrong diagnosis.

That is why safety needs its own metric.

cc: @AnthropicAI @MetaAI @VPrasadMDMPH @HealthcareAIGuy @WillyRontgen @francisdeng @anish_koka @RoupenMD @venkmurthy @zakkohane @arjunmanrai @AdamRodmanMD

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

⚡️Our 3rd Metric (and everyone's favourite): RadLE-A = Accuracy Index.

✅This is the usual benchmark question:

Out of 200 cases, how many did the model get right?

🏆 Top AI models on RadLE-A:

1. Gemini 3.1 Pro — 32.0 👑
2. GPT-5.6 Sol Pro — 29.0
3. Meta Muse Spark 1.1 — 28.5

Human expert baseline: 38.5

Gemini 3.1 Pro was the most accurate AI model and maintains its leadership in accuracy from RadLE 1.0, despite being many months old (We wait for 3.5 Pro eagerly).

Because autonomous diagnosis is not just about getting more cases right. It is also about whether the model knows when it might be wrong.

cc: @GoogleDeepMind @demishassabis @JeffDean @vivnat @alan_karthi @GoogleAI @demishassabis @OpenAI @sama @gdb @MetaAI @Andrejkarpathy @rohanpaul_ai @kimmonismus @haider1 @pushmeet @OfficialLoganK

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

⚡️Our 2nd metric: RadLE-R = Reliability Index.

This asks a very simple question:

When the model gives an autonomous-level answer, is it actually right?

Likert 3–4 = Autonomous-level output.
Likert 0–2 or “I don’t know” = Hand over to human specialist.

🏆 Top AI models on RadLE-R:

1. Claude Fable 5 — 54.7 👑
2. Meta Muse Spark 1.1 — 40.2
3. Gemini 3.1 Pro — 32.5

Human expert baseline: 55.0

This was one of the strongest results. Claude Fable 5 was almost exactly at the human expert baseline on reliability of autonomous-level outputs.

But most models dropped sharply.

So yes, models are getting more accurate.

But their confidence is still not consistently trustworthy.

And if confidence cannot be trusted, autonomy cannot be trusted.

cc: @AnthropicAI @MetaAI @GoogleDeepMind @demishassabis @DrEliDavid @VPrasadMDMPH @hvanspall @AnilMakam @rsumbaly @AIatMeta

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

⚡️The first and primary metric of RadLE 2.0 is RadLE-C = Confidence Weighted Index.

>Correct + confident = rewarded.
>Wrong + confident = penalized.

Because in medicine, a confident wrong diagnosis is not just an error. It is a dangerous error.

🏆 Top AI models on RadLE-C:

1. Claude Fable 5 — 758 👑
2. Meta Muse Spark 1.1 — 735
3. GPT-5.6 Sol Pro — 665

✅Human expert baseline: 988.7

So the best AI is still 231 points behind the average human expert baseline.

🚨The key finding:
The most accurate model was not the top model overall.
Once confidence entered the equation, the leaderboard changed.

cc: @AnthropicAI @MetaAI @OpenAI @karpathy @sama @gdb @tibo_maker @alexandr_wang @Dr_Singularity @rohanpaul_ai @DrEliDavid @reach_vb @thsottiaux @DarioAmodei @jackclarkSF @ThomasScialom @giffmana @kimmonismus @thekaransinghal

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

🔥Today, we are releasing one of the first visual reasoning benchmarks for autonomous AI diagnosis in healthcare!

🚀Introducing Radiology’s Last Exam 2.0 (RadLE 2.0) from @CRASHLabAI, an uncertainty-aware benchmark for autonomous diagnosis in radiology!

✅In the last few days, the AI frontier has moved significantly. @OpenAI launched GPT-5.6 Sol. @Meta launched Muse Spark 1.1. @xAI dropped Grok 4.5.

🙌We’ve benchmarked all frontier, open-source and medical VLMs in RadLE2.0 and the leaderboard is now LIVE!

🚨 Before AI models are handed autonomy, one question matters more than any accuracy score: Do they know when to STOP and hand over to a human?

⚠️ A confident wrong diagnosis is far more dangerous than an honest “I don’t know.” Yet most models are bad at admitting the latter!

🚀 We release five RadLE 2.0 Scores: Confidence Weighted, Reliability, Accuracy, Safety and Handover Readiness and we find that models from @OpenAI @AnthropicAI @MetaAI @GoogleDeepMind @xAI @nvidia @Alibaba_Qwen @MistralAI @MiniMax_AI all score very differently as they optimize for different metrics!

🚨But most importantly, NONE of the Models have been able to reach the average human expert baseline!

⚡️A thread on what we found and which models aced our metrics! Link to the leaderboard and technical report at the end of the thread!

[Piyush @drpiyushENT]

Follow
@CRASHLabAI

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

🔥TL;DR: Radiology's Last Exam 2.0 releasing this week! Stay tuned!

✅People are increasingly using frontier AI models for healthcare diagnosis. And the promise of autonomous AI systems in healthcare has also surfaced recently.

🚨But unchecked social media hype around AI-reported diagnoses is creating a blind trust in models not cleared by any regulatory body.

🤔Just because your AI chatbot sounds confident does it mean it is accurate?

🙌At @CRASHLabAI, we have been actively asking: Are we truly ready for autonomous healthcare systems? With Radiology's Last Exam (RadLE) 1.0 last year, we tested whether the speculation about doctors becoming obsolete actually holds up. Today's AI genuinely works for many common presentations. Where it falls behind are the rare ones!

🔥This week, as the @AIforGood conference commences in Geneva, with world's AI leaders flying in, we are releasing RadLE 2.0 to benchmark the readiness of autonomous AI agents for radiologic diagnosis.

🚨We believe that a model which gives wrong answers confidently is far more dangerous in medicine than one that admits uncertainty! This week, we are going to share with you whether human beings or AI models win this battle. :)

✅RadLE 2.0 is coming… and you will surely want to see these results! Follow @CRASHLabAI and stay tuned!

🇨🇭P.S. If you are in Switzerland for the Summit or around @EPFL or @ETH Zurich for the next 2 weeks, I will be here and happy to meet!

#RadLE #HealthcareAI #RadiologyAI

[CRASH LAB @CRASHLabAI]

This is exactly the kind of story that makes better evaluation important. AI is becoming increasingly capable in common cases. But the question is how it performs on the ones that are uncommon?

That's the question RadLE 2.0 is built to answer. Results out soon!

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

Exactly why we need to release Radiology’s Last Exam v2

Congratulations to Gaurav.

AI models are really good at diagnosing the common and acute conditions. These are 90% of cases. Hence people talk about this on social media more.

But they fail when they move towards rarer conditions, something we call as Out of Distribution (OOD) cases.

We must make the public aware of these limitations otherwise they start believing AI is always correct.

RadLE v2 needs to be out quick. And we are working hard to get this out. More contributors (radiologists) are welcome!

P.S. In next version we plan to expand to all visual domains (beyond radiology).

[CRASH LAB @CRASHLabAI]

“Radiologists will be obsolete. Stop training them.” We’ve heard this since 2016. Radiologists are still here. Broken AI evals are too. Last year, we built RaDLE to benchmark frontier AI. RadLE v1 was our opening move at CRASH Lab @KCDH_A. Stay tuned for our next releases!🧵👇