Ilya Babashin

I build MedEvidence: a clinical assistant that answers from PubMed with every claim cited and checked. Engineer, not a doctor. I publish my evals.

[Ilya Babashin @ILYA_babay]

[1/3]
A 30-second test for any medical AI you use:

Ask it a clinical question. Take one citation it gives you. Look up the PMID.

A year ago half of them wouldn't even exist. Today most do — models got good at returning real citations.

[Ilya Babashin @ILYA_babay]

@MedicalQuack This is the right next layer. A citation tells you a study exists; a quality grade tells you whether to trust it. In my experience, clinicians stop asking "is this cited" fast and start asking "RCT vs case series, and how big." Citation-only starts to feel like table stakes.

[Ilya Babashin @ILYA_babay]

@RoupenMD Where it gets used, not where it looks best - exactly right. For the 60% you're extrapolating on, what would move the needle isn't a higher benchmark score - it's the model showing which paper it's extrapolating from and how far that stretches from this patient.

[Ilya Babashin @ILYA_babay]

@MatthewHellyar Glad it's useful. The main lesson from doing this in ours: making citation mandatory before generation (not a post-hoc check) is what actually kills the fabricated-reference failure mode - worth building in at the retrieval step, not the output step.

[Ilya Babashin @ILYA_babay]

@evidence_md Grounding in evidence instead of the open internet is the right bar for clinical AI. The hardest part in my experience building something similar isn't retrieval - it's making citation a hard prerequisite of the output, not a nice-to-have bolted on after generation.

[Ilya Babashin @ILYA_babay]

@LorencKoka @LinkedIn "Who it's working for" applies to the evidence layer too. A citation next to an AI answer looks authoritative — but n=40 vs n=4,000, industry-funded vs independent, RCT vs retrospective changes what it's worth. Frontline tools should surface that metadata, not just the link.

[Ilya Babashin @ILYA_babay]

@Care_Chronicle Autonomy raises the bar on failure design. An agent that diagnoses and prescribes on its own is only safe if it knows when to stop — abstaining when evidence is thin beats a confident wrong plan. Post-market oversight should measure refusal behavior, not just accuracy.

[Ilya Babashin @ILYA_babay]

@adil_ahmed96 The knowledge base is only half the moat. Retrieval can hand the model exactly the right trial criteria and the synthesis can still drift from them. The eval loop that matters most: does each output claim actually follow from the chunk it cites? Right chunks ≠ right output.

[Ilya Babashin @ILYA_babay]

@pdurdenj This is why audit trails in clinical AI aren't optional. If each AI-generated note traced its medication and date claims to a specific chart entry, the auditor — and the doctor — could catch the error before it becomes a patient safety event. Traceability is the minimum bar.

[Ilya Babashin @ILYA_babay]

85% → 52% is the gap everyone building clinical AI should pin to their wall. And there's a dimension benchmarks don't test at all: citation fidelity. A model can ace the reasoning and still fabricate the reference that justifies it. Real-world accuracy needs right answers AND verifiable sources.

[Ilya Babashin @ILYA_babay]

@HealthcareAIGuy @EvidenceOpen Key nuance: 'source quality' tests whether the right paper was found. The harder test — does the claim faithfully represent what that paper says? Retrieval can be perfect while synthesis quietly drifts. Built around that split: https://t.co/HWQuNHiSLc

[Ilya Babashin @ILYA_babay]

Independent eval is the floor, not the ceiling. The Nature Medicine finding raises a harder question: if general LLMs beat specialized tools on accuracy, what's the remaining moat? Source traceability. Can the model show which PubMed study backed each claim? That's the layer benchmarks don't test yet.

[Ilya Babashin @ILYA_babay]

This is the real stack. Model accuracy is table stakes; the EHR integration + timing layer is where products die or ship. One more layer: even with perfect data pull, does the output trace each claim to a specific PubMed study the clinician can verify? That's what separates 'helps' from 'quietly hallucinates.' https://t.co/QfV3iwLJPD

[Ilya Babashin @ILYA_babay]

@dhruvsuyam Exactly — escalation triggers are the eval nobody scores. Safe behavior isn't a slicker answer, it's the model flagging "this doesn't fit the routine pattern, look closer." Prompt-robust abnormality detection + calibrated refusal beats raw accuracy once it's actually deployed.

[Ilya Babashin @ILYA_babay]

Great thread. The eval methodology gap is real — exam benchmarks miss the clinical reality of ambiguity, coordination, and knowing when NOT to act. One layer both reports likely underweight: does the AI show which PubMed evidence backed each decision, or just give a confident answer?

[Ilya Babashin @ILYA_babay]

@rediminds "Lowers friction without lowering rigor" is the whole game. The friction worth removing isn't generating the answer — it's checking it. If every claim carries its source and is verifiable in seconds, rigor survives the real workflow instead of getting skipped under time pressure.

[Ilya Babashin @ILYA_babay]

@MyuraNagendran The 'two black boxes' framing ages well. What LLMs add: the machine box becomes partly openable — not just via the transcript, but by checking whether each claim is entailed by its cited source. That's the metacognitive check the model can't reliably do for itself yet.

[Ilya Babashin @ILYA_babay]

@MatthewTBishop The 5% stat is the headline. But there's a second gap even inside the evals: MCQ accuracy measures whether the model picks the right answer — not whether its cited evidence supports the claim. Citation fidelity decides bedside trust, and almost nothing measures it.

[Ilya Babashin @ILYA_babay]

📚 Sources (PubMed):
• ESC 2019 PE Guidelines — PMID 31473594
• ADJUST-PE age-adjusted D-dimer — PMID 24643601

Evidence-based symptom analysis → https://t.co/1OwLltB5Wn

#EvidenceBasedMedicine

[Ilya Babashin @ILYA_babay]

🩺 Clinical Case | Pulmonology

Woman, 34. Day 5 after knee surgery. Sudden dyspnea + pleuritic chest pain. SpO₂ 89%, HR 118. D-dimer markedly elevated. Chest X-ray: normal.

❓ Diagnosis? Next diagnostic step?

#MedicalAI #PubMed #Pulmonology

[Ilya Babashin @ILYA_babay]

Dx: Acute pulmonary embolism.

Recent surgery + immobility = classic VTE risk. Tachycardia, hypoxia & pleuritic pain with a NORMAL chest X-ray is textbook PE.

High clinical probability → skip D-dimer, go straight to CT pulmonary angiography (CTPA).