Jason Rothschild

MSK Radiologist | RLHF and Model Evaluation for Medical AI

[Jason Rothschild @JayRothschild]

Enjoyed giving feedback on this one. I agree on the Jevons point. Imaging demand keeps climbing year after year, and a lot of the rules for when a scan is "worth ordering" are shaped by how much the radiologists we have can realistically read, not just what's best for patients. AI expands what's practically possible, and I suspect there's real benefit waiting there.

The new bottleneck is verification. AI hands the radiologist an answer, and checking it carefully isn't much faster than reading the study. I actually think a radiologist reading blind and then comparing with the AI beats one just checking the AI's work, since it's easy to get anchored by a confident wrong suggestion.

Chess went through something similar. For years after engines got strong, human + engine teams beat engines alone because each caught the other's blind spots. Even once AI outperforms radiologists on its own, I think the combination wins for a while. That's the phase radiology is in now.

[Jason Wei @_jasonwei]

AI will not automate radiologists like you think

We have been talking for years now about AI automating jobs. Radiology has been a prime target, with Geoff Hinton once claiming, “people should stop training radiologists now, it’s just completely obvious that within five years, deep learning is going to do better than radiologists”, which is funny since he said that in 2016. There is now more demand for radiologists than ever, so clearly things did not turn out the way many thought. Here I will make a prediction that while radiology will change in the coming decade, radiologists will not be automated away any time soon. I will use radiology as an example but I think these principles will broadly apply to any field where demand can be elastic and quality can scale with increased intelligence.

First of all, we have seen in the short history of language models that there is a “jagged edge of intelligence”, where AI excels at verifiable tasks with a lot of data but is underwhelming on less common tasks. One thing most AI researchers underestimate about radiology is that it is actually extremely multi-task. There are at least a hundred conditions that can be diagnosed from x-rays alone, and thousands using CT and MRI. AI will first master common and easy-to-see conditions and it will not be as good as experts at less-common diseases just because there is less data for that. This jagged edge will certainly improve over time but this means that AI will certainly be used as a triage tool at first, rather than full automation in one go. I would guess that this process from solid triage tool to full automation would take 3-5 years in the best case scenario, because training AI to get all the edge cases right will be bottlenecked by data collection of rare diseases, and also because radiology isn’t currently a focus for frontier labs.

Second, there will be elastic demand for radiology as it gets cheaper per unit. I think elastic demand (“Jevons paradox”) is the most underrated phenomenon when people think about AI automation. In software, it is obvious that I have way more demand than before because Claude Code makes it so easy. These days I ask Claude Code to build a custom three-thousand line html viewer to help me visualize every little RL run that I do. The same thing will happen in radiology. If there is essentially no cost or latency of interpreting an MRI, a physician can order many scans preemptively to better diagnose a case. So with AI tools, a radiologist might have to read only 20% of inbound scans, but there might be 5x more scans inbound. Many developing countries have fewer than one radiologist per million people and they will also want to send a lot of radiology to be interpreted.

Longer term, cheap and fast radiology interpretation will also drive incentives to build cheaper radiology scanners. If an MRI had essentially no latency for interpretation, medical devices companies will want to build much cheaper MRI machines that can be used more frequently, perhaps even in the middle of a surgery. AI will accelerate the development of scanning technology, too. If a full body MRI was only $100 dollars I would totally want to get one every month instead of every two years. I can imagine a world where people get 10 or 100x the number of scans they do today, largely for preventative care, if radiology were 10-100x cheaper.

Third, even if we did have a model that could outperform the world’s best radiologists today, that would not be the end of all work on radiology. Medicine is a field of change. We may develop new treatment protocols that require reading radiology images in a way that radiologists currently aren’t used to. New scanners may reveal diseases that we cannot currently diagnose today with radiology alone. Some radiologists will change their work from reading scans to doing frontier research to advance our understanding of radiology and translate new knowledge into updated models.

Moreover, I am sure that we will ask for more out of our AI models than ever before. With a good flywheel, we may one day train AI models that can use radiology images to predict disease probabilities far into the future. With good prediction algorithms, cancer may be detected months earlier than we can detect it now. Society will invest a lot into this research and this will contribute even more to increased demand for radiology and the best AI around it.

On top of all of this, the medical system is typically slower to adopt new technology and has more regulation compared to technology companies. There is legacy infrastructure and medical centers need to constantly support patients which makes it hard to integrate new technology. This is even more true in developing countries. Moreover data access tends to be limited due to HIPAA, deidentification, IRBs, etc. So I think radiologists will be in demand for a while.

While some worry about radiology being totally automated, I think there is a more exciting future for radiology than ever before—one where there is more demand and we ask for more predictive power from our intelligence. This is similar to how many of us feel that programming will thrive more than ever before despite drastic change in the field. More generally, any domain where demand can be elastic and quality can scale with increased intelligence will, like radiology, similarly flourish with the development of AI. This is certainly a more exciting and nuanced future than what Hinton described in 2016!

(Finally, to be fair to Hinton, he did acknowledge in a recent podcast that he was wrong and also agrees with the elastic demand argument.)

Thanks @JayRothschild and @engyziedan for feedback on this post.

[Jason Rothschild @JayRothschild]

@vvvincent_c That's fair. And may be true of radiology eventually although the data in chess is much cleaner than medical data and the outcomes are much less ambiguous so even though it may get there eventually, the two fields are pretty different.

[Jason Rothschild @JayRothschild]

Enjoyed giving feedback on this one. I agree on the Jevons point. Imaging demand keeps climbing year after year, and a lot of the rules for when a scan is "worth ordering" are shaped by how much the radiologists we have can realistically read, not just what's best for patients. AI expands what's practically possible, and I suspect there's real benefit waiting there.

The new bottleneck is verification. AI hands the radiologist an answer, and checking it carefully isn't much faster than reading the study. I actually think a radiologist reading blind and then comparing with the AI beats one just checking the AI's work, since it's easy to get anchored by a confident wrong suggestion.

Chess went through something similar. For years after engines got strong, human + engine teams beat engines alone because each caught the other's blind spots. Even once AI outperforms radiologists on its own, I think the combination wins for a while. That's the phase radiology is in now.

[Jason Wei @_jasonwei]

AI will not automate radiologists like you think

We have been talking for years now about AI automating jobs. Radiology has been a prime target, with Geoff Hinton once claiming, “people should stop training radiologists now, it’s just completely obvious that within five years, deep learning is going to do better than radiologists”, which is funny since he said that in 2016. There is now more demand for radiologists than ever, so clearly things did not turn out the way many thought. Here I will make a prediction that while radiology will change in the coming decade, radiologists will not be automated away any time soon. I will use radiology as an example but I think these principles will broadly apply to any field where demand can be elastic and quality can scale with increased intelligence.

First of all, we have seen in the short history of language models that there is a “jagged edge of intelligence”, where AI excels at verifiable tasks with a lot of data but is underwhelming on less common tasks. One thing most AI researchers underestimate about radiology is that it is actually extremely multi-task. There are at least a hundred conditions that can be diagnosed from x-rays alone, and thousands using CT and MRI. AI will first master common and easy-to-see conditions and it will not be as good as experts at less-common diseases just because there is less data for that. This jagged edge will certainly improve over time but this means that AI will certainly be used as a triage tool at first, rather than full automation in one go. I would guess that this process from solid triage tool to full automation would take 3-5 years in the best case scenario, because training AI to get all the edge cases right will be bottlenecked by data collection of rare diseases, and also because radiology isn’t currently a focus for frontier labs.

Second, there will be elastic demand for radiology as it gets cheaper per unit. I think elastic demand (“Jevons paradox”) is the most underrated phenomenon when people think about AI automation. In software, it is obvious that I have way more demand than before because Claude Code makes it so easy. These days I ask Claude Code to build a custom three-thousand line html viewer to help me visualize every little RL run that I do. The same thing will happen in radiology. If there is essentially no cost or latency of interpreting an MRI, a physician can order many scans preemptively to better diagnose a case. So with AI tools, a radiologist might have to read only 20% of inbound scans, but there might be 5x more scans inbound. Many developing countries have fewer than one radiologist per million people and they will also want to send a lot of radiology to be interpreted.

Longer term, cheap and fast radiology interpretation will also drive incentives to build cheaper radiology scanners. If an MRI had essentially no latency for interpretation, medical devices companies will want to build much cheaper MRI machines that can be used more frequently, perhaps even in the middle of a surgery. AI will accelerate the development of scanning technology, too. If a full body MRI was only $100 dollars I would totally want to get one every month instead of every two years. I can imagine a world where people get 10 or 100x the number of scans they do today, largely for preventative care, if radiology were 10-100x cheaper.

Third, even if we did have a model that could outperform the world’s best radiologists today, that would not be the end of all work on radiology. Medicine is a field of change. We may develop new treatment protocols that require reading radiology images in a way that radiologists currently aren’t used to. New scanners may reveal diseases that we cannot currently diagnose today with radiology alone. Some radiologists will change their work from reading scans to doing frontier research to advance our understanding of radiology and translate new knowledge into updated models.

Moreover, I am sure that we will ask for more out of our AI models than ever before. With a good flywheel, we may one day train AI models that can use radiology images to predict disease probabilities far into the future. With good prediction algorithms, cancer may be detected months earlier than we can detect it now. Society will invest a lot into this research and this will contribute even more to increased demand for radiology and the best AI around it.

On top of all of this, the medical system is typically slower to adopt new technology and has more regulation compared to technology companies. There is legacy infrastructure and medical centers need to constantly support patients which makes it hard to integrate new technology. This is even more true in developing countries. Moreover data access tends to be limited due to HIPAA, deidentification, IRBs, etc. So I think radiologists will be in demand for a while.

While some worry about radiology being totally automated, I think there is a more exciting future for radiology than ever before—one where there is more demand and we ask for more predictive power from our intelligence. This is similar to how many of us feel that programming will thrive more than ever before despite drastic change in the field. More generally, any domain where demand can be elastic and quality can scale with increased intelligence will, like radiology, similarly flourish with the development of AI. This is certainly a more exciting and nuanced future than what Hinton described in 2016!

(Finally, to be fair to Hinton, he did acknowledge in a recent podcast that he was wrong and also agrees with the elastic demand argument.)

Thanks @JayRothschild and @engyziedan for feedback on this post.

[Jason Rothschild @JayRothschild]

@DrDatta_AIIMS Agreed. It's easy to overlook, but it can cause real issues, and it's something we'll definitely have to be on top of as radiology AI rolls out.

[Jason Rothschild @JayRothschild]

@_jasonwei Thanks for including me, Jason. Great piece!

[Hyung Won Chung @hwchung27]

In a lecture I gave last year (https://t.co/gyBq2vRbvD), I shared a mental model for AI: the convex hull of humanity’s collective knowledge.

Human expertise is highly specialized and fragmented across people and fields. I hypothesized that there might be a huge overhang of new knowledge accessible through “interpolation”: connecting the dots in this high-dimensional knowledge space.

@_jasonwei and the team are showing what connecting the dots can look like with an existing trove of health data. Exciting to see AI helping researchers turn that data into new scientific insights!

[Arturo LoAIza-Bonilla, MD MSEd @DrArturoAI]

Absolutely thrilled to have been able to use Muse Wonder and be part of this scientific research amazing launch! 🚀 https://t.co/Qb5VvOP2k2 @finkd @alexandr_wang @Meta @mystlukes

[Jason Wei @_jasonwei]

muse wonder is out! scaling prevails…

it has been so fun to be part of the powerful yet nimble ship of Meta, where in just over a year, much of the company pivoted to contribute to the model, ranging from the infra engineers who started building tools for experiment management to the business development teams that totally transitioned to help hire human experts for data review

and try out our model for health use cases as well! great for multimodal, you will be able to feel the scale 📈

[Jason Wei @_jasonwei]

In our health workstream at Meta we did a project where we wanted to see what research papers medical experts could write using a Muse Wonder coding scaffold that could access a database of 5M electronic health records we licensed from a vendor. The goal is that with 5M patients in this database, AI could do research to answer arbitrary questions like “did people who take statin have higher rates of diabetes later in life”? The thing we were betting on here is on what happens when you put human experts, a strong coding model, and access to data all together

We worked with a small group of medical experts to help them use this AI tool and the punchline is that they all found that this setup enabled a much faster pace of health research. They generated ten manuscripts in around two weeks, with one researcher (Paddy Ssentongo) generating three. Throughout the process, they collaborated with each other to review manuscripts and write skill files to help our coding scaffold improve at doing research. The medical experts liked how quickly they could test different hypotheses and choose the best ones to continue working on

Health research is pretty unique in that it is super important to be able to interpret noisy data and have taste to know which variables are important, which was still a significant value add from human researchers. Given that medicine is one of the oldest fields in human history and it is ultimately about humans, keeping a substantial human component in this type of research is probably important. In the near future I see AI primarily as a tool to facilitate human-led research. With the right scaffold I believe health researchers can be 10x or even 100x more productive using AI tools that have appropriate access to the wealth of data that the world has collected about human health

One way to see this work is through the broader lens of AI for science. You can think of science as two parts: (1) collecting raw data about the world, and (2) synthesizing that raw data into understanding. In this project we already had existing data about the world and we aimed to accelerate part (2) using AI. Eventually, I imagine a world where we have such great AI tools that the time to synthesize raw data into understanding is driven towards zero. In that world, science will be substantially faster and most of the work will be focused on collecting the right raw data via physical experiments

We did this in a pretty short amount of time (which was the point) and I think the future of scientific research will be more faster and more exciting than ever

[Jason Rothschild @JayRothschild]

Really proud to be part of this group and of what everyone built in a few weeks.

I'm amazed how much it's just about the thinking now. I can have an idea and test it without all the tedious speed bumps in between. Research feels purer.

Excited to see where this goes!

[AI at Meta @AIatMeta]

We’re exploring how AI can accelerate health science research.

While AI has made significant breakthroughs in formal sciences like mathematics, health data science depends on imperfect real-world data and consequential choices about cohorts, variables, and causal assumptions.

To explore how AI can assist health researchers, we built a human-led research scaffold powered by Muse Spark 1.3 and Muse Wonder with access to 5M retrospective de-identified electronic health records and publicly available data. A cohort of medical experts used the system to conduct original research, accelerate hypothesis generation, code execution, and manuscript drafting while maintaining full authority over scientific decisions.

Today, we’re sharing 10 research manuscripts produced through this exploration, spanning cardiology, oncology, infectious disease, emergency medicine, and more.

Read more in our research blog: https://t.co/26l6Du6vIk

[Jason Wei @_jasonwei]

Love this quote about Muse Wonder:
"What I appreciate about Muse Wonder is that it isn't sycophantic. Some other models I've used will make an idea sound great in the moment, and only later do I realize it didn't hold up. Muse Wonder is willing to call me out and steer me toward a better approach, often before I've spotted the problem myself. That saves a lot of time, and it's a big part of why I trust it." — Jason Rothschild, MD, Musculoskeletal radiologist

[Jason Rothschild @JayRothschild]

@CardoneMD @northwoods1980 Completely agree. System needs to reward this behavior with legal protection / money. Weird how we still don't have anything like this.

Think AI could be interesting at a certain point - AI gold standard that would protect you legally if you follow its rec to not order a test.

[Jason Rothschild @JayRothschild]

@NealBruceBC That sounds great! Are you reviewing your lichess opening studies with chessable?

And if so, have you figured out an effective way to sync the lichess study updates with chessable?

[RJ @northwoods1980]

All of these middle aged patients still coming through orders with “trochanteric bursitis“. Just stop it. You know it’s usually anterior gluteal medius chronic avulsive tears right!😩. It’s hardly ever bursitis

[Jason Rothschild @JayRothschild]

@yan_yet @northwoods1980 Interesting. Do you use a different method?

[Jason Rothschild @JayRothschild]

@JRevRad1 @RadioGraphics @ShamusMoran @BaharMansooriMD @MargaritaRevzin @moshirimd @DaveDisantis That was awesome! Thank you.

[Jason Rothschild @JayRothschild]

@northwoods1980 Do you think of transient osteoporosis as a similar process?

[Jason Rothschild @JayRothschild]

@askavrukh I think it's called horde chess

[Jason Rothschild @JayRothschild]

@northwoods1980 Asymmetric soft tissues in the left medial thigh

[Christian Barrera, MD @CBarreraMD]

Imaging findings in ischemic stroke 🧠

This is a more pragmatic approach. There are several classifications in the literature, but most of them follow this sequence of events

Thanks to @francisdeng for his feedback!

#radres #radiology #MedTwitter #FOAMrad #Neurology #Neurorad