Kirk Willmarth

Healthcare consulting by day, but at heart just a nerd for science news and health policy.

[Ron Alfa @Ronalfa]

We're excited to share TARIO-2, a new bio foundation model architecture for generative spatial whole transcriptomes from simple pathology images!

Read more here:

[Alex Prompter @alex_prompter]

This paper from Harvard and MIT quietly answers the most important AI question nobody benchmarks properly:

Can LLMs actually discover science, or are they just good at talking about it?

The paper is called “Evaluating Large Language Models in Scientific Discovery”, and instead of asking models trivia questions, it tests something much harder:

Can models form hypotheses, design experiments, interpret results, and update beliefs like real scientists?

Here’s what the authors did differently 👇

• They evaluate LLMs across the full discovery loop hypothesis → experiment → observation → revision
• Tasks span biology, chemistry, and physics, not toy puzzles
• Models must work with incomplete data, noisy results, and false leads
• Success is measured by scientific progress, not fluency or confidence

What they found is sobering.

LLMs are decent at suggesting hypotheses, but brittle at everything that follows.

✓ They overfit to surface patterns
✓ They struggle to abandon bad hypotheses even when evidence contradicts them
✓ They confuse correlation for causation
✓ They hallucinate explanations when experiments fail
✓ They optimize for plausibility, not truth

Most striking result:

`High benchmark scores do not correlate with scientific discovery ability.`

Some top models that dominate standard reasoning tests completely fail when forced to run iterative experiments and update theories.

Why this matters:

Real science is not one-shot reasoning.

It’s feedback, failure, revision, and restraint.

LLMs today:

• Talk like scientists
• Write like scientists
• But don’t think like scientists yet

The paper’s core takeaway:

Scientific intelligence is not language intelligence.

It requires memory, hypothesis tracking, causal reasoning, and the ability to say “I was wrong.”

Until models can reliably do that, claims about “AI scientists” are mostly premature.

This paper doesn’t hype AI. It defines the gap we still need to close.

And that’s exactly why it’s important.

[Kirk Willmarth @kirkwillmarth]

This is a worthwhile and wonky read on some of the potential (existential) impacts to the FDA based on recent events

[Alexander Gaffney @AlecGaffney]

This piece is now available to read on our free site due to public interest: https://t.co/Qff2lqiFeg

[Kirk Willmarth @kirkwillmarth]

@LizzyLaw_ Interesting that AI in devices seems directly targeted. Any sense for whether the impacted people were involved in Neuralink interactions?

[Larry Levitt @larry_levitt]

Starting today, Medicare beneficiaries will have their annual out-of-pocket drug costs capped at $2,000 as a result of the Inflation Reduction Act.

[Ron Alfa @Ronalfa]

Announcing OCTO-VirtualCell (vc) a multi-scale, multimodal transformer trained to predict gene expression for a virtual cell in cellular contexts within patient tissue samples. Complete wth the Celleporter demo app to explore the data!

1/

[Berna Sozen @BernaSozen_]

Our latest out today @Nature!
We revisit a century-old theory and show that metabolic gradients work hand-in-hand with genetic&signalling instructions to shape the embryo. Glucose isn't just an energy source, it acts as a major conductor of the body plan🍬
https://t.co/mfx6Hk7nma

[Mushtaq Bilal, PhD @MushtaqBilalPhD]

50% of scientists stop publishing within a decade of starting their career.

A recent study looked at the careers of 140,000 scientist and concluded that less than half were still publishing 15 years later.

[Kirk Willmarth @kirkwillmarth]

I had this exact experience. Our automated rent increase put us over the available open unit down the hall. We asked them to match, they said they were not allowed to, so we moved units for a lower rent, and even got a new tenant incentive on top (gift card).

[Scott Santens @scottsantens]

Housing by algorithm is so much nonsense. I'm experiencing it directly in a way I hadn't known about before. My apartment building uses one of these legal price-fixing algorithms. Our lease is up soon and we saw that identical units to ours were going for much cheaper than what we were being offered to stay in our unit.

We told them we'd be happy to stay if they lowered our rent to the market price, but they said their hands were tied. They must obey the Algorithm. The almighty Algorithm appears to mandate that all rent must rise if the current renter wants to renew a lease. The increase must be at least ~1.5% even if a new renter will pay much less.

So we decided to move into a unit slightly larger than our existing unit and pay $600/mo LESS. Our existing unit is now on the market for $600/mo less too. We would have preferred to just stay where we are and pay less, but Our Lord Algorithm does not allow that.

It's weird to think that what we could have done was not renew our lease, and then sign a new lease in the same unit the minute it went on the market for less. But we weren't allowed to just renew our lease at the lower rate. Because of the Algorithm.

So keep that in mind, fellow renters. When your lease is up, make sure and check to see what the Algorithm God is saying the rent should be for similar places, not what it says you should pay for your current place. And consider the savings of moving within the same building if that's an option, or even risk not renewing and picking up your same place with a lower market rate lease.

All hail the Algorithm.

[Spencer Greenberg 🔍 @SpencrGreenberg]

Does astrology work? We tested the ability of 152 astrologers to see if they could demonstrate genuine astrological skill.

Here is how the study was designed and what we found (including a result that really surprised me):

🧵

[Scott Gottlieb, MD 🇺🇸 @ScottGottliebMD]

SCOTUS decision is very significant for FDA. Courts will continue to defer to FDA on product-review decisions, where Congress gave FDA discretion to make fact-based decisions based on careful process often outlined in guidance. But we'll see material changes in other areas 1/n

[NCHS @NCHStats]

#STATOFTHEDAY The overall number of Americans without health insurance dropped by 8.2 million from 2019 to 2023 https://t.co/4jf6zz3Kgm

[STAT @statnews]

There was mounting excitement about the FDA's review of Lykos' MDMA-assisted therapy. But former employees said the company committed a series of lapses. https://t.co/ahLmvgaYUp

[Paras Sharma @paras_biotech]

VC Fund size to DPI plot - 168 funds covered & Vintage year 2014-2016

1. Only ~20% are in the carry
2. >$2B barely returned deployed capital back to LPs (not including fees/hurdle)
3. Median fund size of $200M returned >1.5x DPI

Will post other thematic observations later

[NYT Science @NYTScience]

Daniel Kahneman helped pioneer a branch of economics that exposed hard-wired mental biases in people’s economic behavior. The work led to a Nobel. He has died at 90. https://t.co/KTfB1XBLFh

[Ethan Mollick @emollick]

Write up of our paper: https://t.co/MqXtAdn3bh

[Eric Topol @EricTopol]

When #AI support was provided to 140 radiologists, there was marked variability and unpredictability as to its impact on performance
https://t.co/YGXUcagIWi @NatureMedicine @pranavrajpurkar @feiyangkathyyu

[Carlos D. Bustamante 🇻🇪🇺🇸 @cdbustamante]

We at @GalateaBio are thrilled to be partnering with @illumina to build the Biobank of the Americas, a 10 million person participatory cohort to enable precision health at scale for all.
https://t.co/JpakUj9U8y

[Bruce Booth @LifeSciVC]

Crowding is happening: HER2 has 14 active programs in the clinic, FRa and CLDN18.2 have 7 each.

[Larry Levitt @larry_levitt]

New: Somewhat remarkably, just 39% of adults know that the Affordable Care Act prohibits insurers from denying coverage due to pre-existing conditions.

https://t.co/kbxSnxdCgE