Adam Rodman

Physician, educator, historian, author, podcaster, researcher @BIDMC_IM @HarvardMed @HarvardDBMI, host of @BedsideRounds, AE @NEJM_AI, studies 🤖+🧠. 🖖🚲

[Adam Rodman @AdamRodmanMD]

In case you were wondering why there hasn't been a new @BedsideRounds in a while ...

https://t.co/M5cZvTmFwJ

[James Zou @james_y_zou]

How long does a finding about LLMs stay true?

We re-tested 50 widely cited findings about LLM behavior (various biases, hallucination, etc). Within 15 months, most no longer held or had substantially reduced effects.

LLM findings has short shelf life https://t.co/MUWm6INyX8

Great job by @oq_35!

[Adam Rodman @AdamRodmanMD]

@irenetrampoline @LaudeInstitute Congratulations! This is absolutely huge AND so important!

[Google for Health @GoogleForHealth]

Introducing the Google AI for Healthcare Professionals Certificate, practical training developed by clinicians at Google to help healthcare professionals use AI responsibly in their daily work.

[Kashif Pirzada, MD @KashPrime]

I grew up in a magical era when the leading man of the leading TV show at the time solved hard problems with intelligence, courage, honour and respect. I owe so much to his example, and continue to try to live by the example his character set.

[Wes Boland, MD @Wes_boland]

Increasingly important as we seek AI-clinician alignment is cost. Across 20 LLMs, AI-suggested care cost a mean of $314 per visit, compared with $97 for care recommended by treating physicians. Specifically driven by extra lab testing.

https://t.co/8ioO8R3ruS

[Adam Rodman @AdamRodmanMD]

@DShaywitz @NEJM @Gurpreet2015 @sanjaysaint @Bob_Wachter (we have some similar working coming out -- lots of exciting work to be done to make sure these tools can assist in diagnosis!!!) Many congrats to the Microsoft team.

[Adam Rodman @AdamRodmanMD]

@DShaywitz @NEJM @Gurpreet2015 @sanjaysaint @Bob_Wachter MIMIC-IV is not the ideal dataset either to test this -- but I think it definitely gives directional information.

I think the fundamental lesson of this paper is: you get better performance by effectively getting LLMs to "roleplay" information theory.

[Adam Rodman @AdamRodmanMD]

@jasongusdorf @bratogram @bethisraellahey The feeling is mutual Dr. Gusdorf!!

[Adam Rodman @AdamRodmanMD]

I can't believe it's really happening! For the last year, @bratogram and myself have been putting together a brand new AI Center at @bethisraellahey that we hope will serve as a catalyst across the entire system for advancing the responsible study and adoption of AI in medicine.

[BIDMC Smith Institute @SmithBIDMC]

We are thrilled to announce the Center for Surgical and Medical Intelligence at the Smith Institute! Led by @AdamRodmanMD and @bratogram, the Center will advance the science & capabilities of #ArtificialIntelligence in medicine & surgery: https://t.co/KN7KHyB6Id

@bethisraellahey

[Adam Rodman @AdamRodmanMD]

I could not be more excited to be a part of the @SmithBIDMC, and to work with our clinicians, research colleagues, industry partners, and most importantly our patients.

Medicine moves at the speed of trust, and we are going to bring that clinical grade evidence to AI.

[Adam Rodman @AdamRodmanMD]

We are focusing on building durable infrastructure to advance @bethisraellahey's and @BIDMChealth's clinical and research missions, and ensure that advanced AI systems are deployed in ways that are safe, effective, and enhance the humanity in medical care.

[Liste @endliste]

Just found out about recency bias. Maybe the most important bias ever

[Arvind Narayanan @random_walker]

We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"

[Iain Cameron @theiaincameron]

Delaware’s northern border with Pennsylvania is one of the strangest state boundaries in America. Instead of following a river, mountain, or straight survey line, much of it is an arc of a circle.

The story of the circle's centre is wonderfully quirky...

1/n

[Adam Rodman @AdamRodmanMD]

@leah_pierson @Maratosflier We have the weirdest Spotify wraps ...

[BIDMC Research @BIDMC_Research]

Congrats to @BIDMChealth @AdamRodmanMD and team whose @Lancet study with @Google researchers is the first prospective real-world test of a patient-facing AI chatbot in primary care. Read more: https://t.co/t6njpce5w1

[Vivek Natarajan @vivnat]

Delighted to see our real-world clinical study of a conversational diagnostic AI system published in @TheLancet, alongside coverage in @nytimes last week.

Looking back, AI in medicine might feel inevitable but it actually takes deliberate, uncertain leaps to get there. I have been fortunate to be part of two such moments.

The first was early 2022 (pre ChatGPT) when @alan_karthi and I got access to our first @Google LaMDA and PaLM models. That convinced us to go all in on LLMs in medicine, sparking the Google Brain moonshot leading to Med-PaLM.

The second moment leading directly to this Lancet study, was late summer 2023.

@taotu831 showed me the early AMIE models he was training with our self-play loop, targeting our upcoming OSCE study with patient actors, later published in @Nature 2025.

At that exact time, I was navigating a personal health condition. Testing myself, the consultation the AI performed was so natural and clinically sound that I was convinced it was ready for the real world.

So I reached out to Alan and together with the team we asked how we could test this in the real world quickly, safely and responsibly. We needed a prospective clinical trial with a safety endpoint and all roads led to @AdamRodmanMD at @bethisraellahey @harvardmed .

Adam was incredibly gracious with his time kicking off our first discussions in the fall of 2023.

But clinical trials take a long time; years of painstaking work. There are so many critical people and moments that make them happen, far beyond the glitzy PR materials and the main authors.

- For example, a discussion with @greg_corrado sparked the design of the oversight setup underpinning the safety layer in the study.

- Kavita Kulkarni (now at BioHub) spent months negotiating the initial contracts with BIDMC; it was uncharted territory for both organizations.

- @_khaledsaab (now rocking at OpenAI!) working tirelessly to improve the AI, together with @taotu831, @apalepu13, @RyutaroTanno, @ckbjimmy, @valentinlievin, @elahevedadi and many more folks.

- @Mysiak, Alessio Orlandi, Petar Sirkovic and team sharing the seamless serving infrastructure they built for Co-Scientist, which we could re-use for much of this study.

- @SSaraMahdavi making sure the study team was well organized :)

This is just to name a few. Prospective trials are a massive team sport and I am proud of what our teams achieved.

While the AI's clinical reasoning was strong, seeing the patient trust, safety, and satisfaction data, with ZERO safety stops from our supervising physicians, is what truly stands out.

At the same time, this is still a stepping stone. We are moving from AI as a tool to an active care team member - a guardian angel amplifying clinicians and staying with patients throughout their health journey.

More to come soon, and as always, a pleasure to work with so many great folks across @GoogleDeepMind , @GoogleResearch and @bethisraellahey .

Paper - https://t.co/Mmp0D7RluN

NYT - https://t.co/Wg7nQCoTXo

[Adam Rodman @AdamRodmanMD]

@MattZirwas @arjunmanrai something that doesn't matter.

[Adam Rodman @AdamRodmanMD]

@MattZirwas @arjunmanrai Raj is referring to the experimental (and HCI) literature, which is much more akin to some of our famously unreproducible psych experiments (ahem, power posing ...).

I only like 50% agree. Those elements are present in plenty of the studies ... but we're also often measuring ...

[Adam Rodman @AdamRodmanMD]

@AdamRodmanMD @arjunmanrai Yes, replication crisis implies there was something to replicate.

Until AMIE there was not.

[Adam Rodman @AdamRodmanMD]

Our clinical trial of AMIE, a patient-facing AI system, was just published in The Lancet.

I want to explain what *I* think this study ACTUALLY means.

It's about a fundamental question: how do we know that clinical AI works?

🧵⬇️

https://t.co/5Ue2CrjSpY

[Google @Google]

Our medical research system, AMIE, is the first patient-facing conversational diagnostic tool of its kind to be studied prospectively in a real-world clinical setting.

In a study, published today in @TheLancet, we found that when patients chatted with AMIE before an in-person appointment, the interaction built their confidence and helped them organize their thoughts. And physicians who reviewed the information from the patient conversation got more time back to focus on collaborative care and shared decision-making instead of digging through data.