Jiahan Li

PhD @NYU_Courant w/ @rob_fergus & @kchonyc, SR @GoogleResearch, Previous: @pku1898 Working on Generative Models, AI4S

[Jiahan Li @JiahanCed]

Why do most unified models still stitch together two different paradigms: an autoregressive LLM for text, and an attached diffusion head for images? (1/16)

We wanted to see if one generative process could handle both modalities on equal footing.

We introduce JUICE 🧃 (Joint Unified multimodal modeling In Continuous Encoder spaces): a flow-matching Transformer (MM-RiT) trained from scratch over frozen pretrained image and text encoders, with no LLM initialization.

Each 2.1B model uses one set of weights:
- SigLIP2 variant: 0.87 GenEval, 85.27 DPG-Bench, 0.45 WISE, 73.1 VQAv2
- DINO variant: 0.88 GenEval (tying TUNA, the strongest 1–3B baseline)
- Parallel text decoder: 3.9–6.4× faster than AR on VQA, 54–163× on captions

Blog: https://t.co/Kq3m5QksMH
Paper: https://t.co/9E5LWVJCKA
Weights: https://t.co/iIM9FjtJ8P
Code: https://t.co/efBAPow42j

[Professor Oak @Prof_Oak_]

Surprise 1: Anthropic acquired a one-person biotech
.
.
.
Surprise 2: that person is me
Surprise 3: I’m here to tell you all about it

Yes, this is what you think it is; after 6 yrs of increasingly loose anonymity: the Professor Oak reveal post

…and yes, it has gotten so boring around here that I’m literally doxxing myself to spice up the timeline

-------
Ok, allow me to introduce myself. My name is Cody Tranbarger. I mostly tweet, but occasionally find time to dabble in drug dev & starting/building biotechs. My past lives span the VC, crossover, startup continuum; Prof Oak has been my way of keeping a digital foot in those worlds, despite having my physical feet planted elsewhere.

The past 18 mo, ‘elsewhere’ was Mithril Medicine, a co. I founded to develop bsAbs for LAMA2-CMD & other ECM-related genetic diseases.

Mithril was born out of hands-on building with @MikeNGladstone, @JasonPaulRhodes, @LifeSciVC, & team @atlasventure. Somehow, I made enough half-decent calls along the way (even a broken clock is right twice a day, eh?) to convince them Mithril/I warranted a small seed. While I can hardly rationalize such an outrageously cavalier bet, I’m grateful for it, & I'd partner with Atlas again in a heartbeat.

In typical AV fashion, Mithril’s primordial form was virtual, decentralized, & ultra-lean. This stage tends to be short-lived, but for Mithril, it worked surprisingly well, for surprisingly long, thanks in part to the rapid maturation of the AI tools I used liberally.

Eventually, Mithril became a bit of an experiment itself - to find the upper bound of how far it could scale as a single operator + a bunch of Claude (+ a stellar bench of part-time consultants).

In parallel, I started spending time with @nc_frey, @an1lam, & @CoefficientBio team; and promptly experienced, for the 1st time, AI at the frontier. Soon thereafter, Mithril stopped looking like an experiment, & started looking like a preview of an entirely new operating model.

From that spark came “Pipeline-in-a-Person.” The idea immediately resonated when we shared it here back in April; and it was central to the shared vision that led to Anthropic’s acquisition of Mithril, & my joining the team.

Why join Anthropic? Three reasons.
(1) For Mithril, the programs continue, with more resources & infrastructure behind them, and as much Claude as I can put to work.

(2) For me, the chance to start several more programs, & find out how many one person can actually run in parallel - an almost unprecedented luxury for a curious drug hunter.

(3) For our mission, a path to institutionalize & scale this new way of working, then put it in the hands of all of biopharma, all over the world - which, candidly, I find the most energizing.

Will AI x Bio cure diseases? Unequivocally yes, imo… but not by the route that gets the most airtime. Rather than timeline acceleration & compression, I see orchestration & parallelization as today's low-hanging fruit.

Most drug hunters see a handful of go/no-go calls in a career, & spend entirely too much of the rest trading emails (my inbox can attest).

Give them the bandwidth to run several programs at once, and the time to focus on scientific judgment, and I’m willing to bet they’ll make better calls. And at scale, better calls translate to more drugs, higher hit rates, broader patient impact, & likely, a disease or 10 or 100 cured. I couldn't be more excited to build toward that future @AnthropicAI.

Finally, a heartfelt thank you to BioTwitter/X. The end of anonymity is bittersweet, but if anyone’s earned a proper introduction, you all have.

It’s difficult to overstate how much this app has done for me: a decade of education, plenty of friends, many job opportunities, several mentors, a few investors, even a co-founder (fmr). It was the generosity of this community that inspired me to start posting 6 years ago, after just as many years lurking. I hope I’ve been able to pay it forward, in my own small way.

But, to be clear, Prof Oak isn’t going anywhere. Same beat, same themes, and now, a front-row seat to drug development at the frontier. Stay tuned for dispatches from the trenches; and remember...

Gotta Cure 'Em All

[Jiahan Li @JiahanCed]

That's crazy.

[Anthropic @AnthropicAI]

Over the past few months, Anthropic scientists have engaged in deep research partnerships with six organizations developing new medicines. We've also begun drug discovery efforts of our own.

In doing so, we’ve learned much more about how Claude can support the many stages of drug development.

In this blog post, we share some early results from these efforts, and describe how this work fits in with our wider ambitions in the life sciences.

https://t.co/EVbxkdDTgJ

[Minhyuk Sung @MinhyukSung]

Does every multimodal AI model need a massive LLM at its core?

Diffusion language models are gaining momentum. But diffusion/flow models could play an even broader role: enabling multimodal generation without putting language at the center.

Our MUNI & MUNITE projects explore this direction. 🧵

[Zizhang Li @zizhang_li]

🌍 Actions that condition world models are highly heterogeneous: forces on objects, camera movements, hand motions, robot arms, each with its own meaning and structure. So today's action-conditioned world models are usually built one action type at a time, each with its own conditioning design.

We introduce UniAction, a unified action-conditioning framework: one world model driven by all of these actions, and even by mixes of them in a single generation. Trained jointly, it outperforms the action-specific experts. Great work led by amazing @BowenXue_ .

More detailed threads below👇
🧵 1/10

[Jiahan Li @JiahanCed]

🔥🔥🔥

[Kyunghyun Cho @kchonyc]

Ortet's mission is to help create healthcare where everyone can receive predictably better care. in pursuing this mission, we believe predictability must come together with transparency and controllability.

taking the first step in this direction of transparency and predictability in health, we have created the Orion cluster, host a large fleet of open-weight models and began our evalution campaign on health-focused benchmarks.

in doing so, we learned a lot about building up an agent-first cluster and also learned about the current state of AI models in the context of health benchmarks. we are excited to share the first set of findings from this ongoing campaign on our Orion cluster.

[Morgan Levine @DrMorganLevine]

I've been meaning to share these thoughts for a while and touched on them briefly in a recent conference talk. They are relevant to a lot of the discussions today. https://t.co/YywkAS87eQ

[Diego del Alamo @DdelAlamo]

@RolandDunbrack My current mental model of ipSAE is that the low end (depends on the target) is extremely predictive of nonbinding, but higher values are not. So it is useful but not perfect at filtering. Per @amirshanehsaz's comment, 11/60 ain't bad!

[Jimmy Heaters @CathPoaster]

If you think San Francisco is only 4-5 years behind NYC culturally then you are a philistine idiot. San Francisco is decades behind NYC and anyone with an iota of taste in their body knows this. NYC clears SF in food, night life, music, arts, fashion, urbanism, walkability, intellectual spaces, and exposed brick walls. If you have observed how people in the two cities even just dress and carry themselves you know it's a hydrogen bomb vs coughing baby situation. NYC has comedy clubs, theater, and intellectual life spanning EVERYTHING, not just AI and tech. In New York I can go see a play, get dinner in a dimly lit restaurant, and then get drinks with my friends for like $500. You can't do that in SF. In NYC I can go to a party in Brooklyn and meet amazing experimental visual artists who really don't like getting asked if their parents pay for their nice 1 bedrooms. Yesterday I went bar hopping during the day to catch all the Big 10 football games which is becoming a huge cultural mainstay in Manhattan. Yeah, we're culturally and intellectually superior to San Francisco tech nerds. Do I even need to mention the subway?

[Jiahan Li @JiahanCed]

@Pandatree537201 Thanks for suggestions! I'm curious why your text has so many em dashes🥹😂

[Jiahan Li @JiahanCed]

Why do most unified models still stitch together two different paradigms: an autoregressive LLM for text, and an attached diffusion head for images? (1/16)

We wanted to see if one generative process could handle both modalities on equal footing.

We introduce JUICE 🧃 (Joint Unified multimodal modeling In Continuous Encoder spaces): a flow-matching Transformer (MM-RiT) trained from scratch over frozen pretrained image and text encoders, with no LLM initialization.

Each 2.1B model uses one set of weights:
- SigLIP2 variant: 0.87 GenEval, 85.27 DPG-Bench, 0.45 WISE, 73.1 VQAv2
- DINO variant: 0.88 GenEval (tying TUNA, the strongest 1–3B baseline)
- Parallel text decoder: 3.9–6.4× faster than AR on VQA, 54–163× on captions

Blog: https://t.co/Kq3m5QksMH
Paper: https://t.co/9E5LWVJCKA
Weights: https://t.co/iIM9FjtJ8P
Code: https://t.co/efBAPow42j

[Jiahan Li @JiahanCed]

@JinWeiyang18434 Thanks sharing! Well I still think we need someway to inject world knowledge from I’ll to these generative models / Or maybe adding heavy uni-modal language data.

[News from Google @NewsFromGoogle]

Gemini 4 Argon is already helping researchers unlock new scientific breakthroughs across fusion, biology, and quantum computing:

⚛️ In nuclear fusion, Argon helped discover a broader class of counterexamples to Grad's conjecture — a nearly 60-year-old assumption in plasma physics — creating new benchmarks for stellarator 3D magnetic fields.

🧬 In biology, it designed five novel, lab-verified serine esterases in a matter of hours — enzyme engineering that would previously have taken months. This work demonstrates the potential for rapid acceleration in fields such as biomanufacturing, green chemistry, and drug development.

🩺 In rare disease research, scientists at @GamowLabs used Argon to analyze de-identified genomes from newborns with rare developmental lung disorders — pinpointing a chromosomal rearrangement that routine software missed for years, and explaining why it went undetected.

🧮 In quantum computing, Argon translated our AlphaQubit error-correction decoder into a single optimized hardware kernel — nearly doubling inference throughput — and cut spacetime costs for a key quantum subroutine by more than 100x.

[Jiahan Li @JiahanCed]

@swe_acc lol we should shout out to Unidiffuser https://t.co/C6x6puSQdG

[Jiahan Li @JiahanCed]

@owengretzinger Thanks!

[Jiahan Li @JiahanCed]

Thanks ! We empirically use the same distribution for images and text; for example, for images we use the distribution from the RAE paper.
I think we could sweep time distributions to find a better config, but the original distributions already work very well (and widely used in different papers lol).

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

What if multimodal generation didn’t have to choose between understanding and creating, or between images and text going first?

Excited to share JUICE 🧃 with @JiahanCed and the team!

What I find especially interesting is the role of representations: the same features that the model uses to understand an input also become the states it learns to generate. Images and text can then take shape together, with each modality providing context for the other throughout generation.

One detail from our ablations: even when only one modality is supervised on joint examples, keeping the noisy partner stream improves image generation over conditional-only training at the same target count. Cross-modal interaction can be useful before either side is fully formed.

The maze experiments make this tangible. A drawn route and a written sequence of moves can each be valid while describing different solutions. Coupled generation lets the two streams settle on a consistent pair, without choosing in advance which modality goes first.

This also raises an interesting question for multimodal memory: could similar interactions help models recover useful context from incomplete visual and linguistic cues?

Excited to explore where this direction leads!

[Jiahan Li @JiahanCed]

@kartikb753 Totally agree! Maybe we need better language or multi-modal encoder? Or we need to add more uni-modal language data during training.

[Jiahan Li @JiahanCed]

@RyanJ_KOL Yes, we include some results in the paper's appendix.
Honestly, I still feel there's a long way to go for continuous diffusion language models; in this paper, we show that modeling language in continuous space helps a lot with multimodal tasks.

[Jiahan Li @JiahanCed]

Thanks @Huangyu58589918 @TongPetersb @shushengyang for endless discussions and support, @RuihanGuo2 @hungsuzh143318 @zhaohexu2001 for fun ideas and advice, @pratyusha_PS @sherryyangML @thoma_gu for guidance, and above all my advisors @rob_fergus @kchonyc for always backing me.

[Jiahan Li @JiahanCed]

Results 4/5: joint reasoning on mazes.

On synthetic mazes with 2–16 shortest routes, our task-adapted 2.1B SigLIP2 model (224×224) draws a route and writes its moves; both must be valid and match (93.7% coupled vs 54.6% independent):
- Independent sampling: 54.6%, as each stream often picks a different valid route
- Text first / image first: 85.3% / 95.2%. The more reliable modality should go first, but on a new task we don't know which one that is in advance.
- Coupled sampling (both streams updated together): 93.7%, within 1.5 points of the better fixed order without choosing one

Generation order is no longer hardwired: it becomes a sampling-time choice, and coupled sampling can leave it open. In the two traced mazes, whichever output settles later adapts to match the other.

[Jiahan Li @JiahanCed]

Results 5/5: planning by drawing and writing.

On visual spatial planning (VSP, FrozenLake), our task-adapted 2.1B SigLIP2 model (224×224, trained on synthetic maps) draws the path and writes the moves together with coupled sampling:
- Training grid sizes (3×3 to 6×6): 99.8% success, on par with LatentUM-fine (99%). LatentUM-coarse reaches 85%, ThinkMorph 76% and InternVL3.5-38B 20%.
- Unseen larger grids: 94% on 7×7 and 69% on 8×8.

On unseen sizes, the drawn path is more reliable than the text moves, matching the wording cost from Takeaway 1. On 9 of the 600 benchmark maps, the moves initially fail and are corrected during coupled sampling; in the 8×8 map shown, they first walk into a hole.

[Jiahan Li @JiahanCed]

Results 3/5: an image and a caption from noise, together.

With no condition, both streams start from noise and we generate an image and a caption at once. Because the text clock runs ahead, the caption takes shape before the image does, while each stream conditions on the other at every step. In the selected samples, each caption describes the image generated alongside it, even though neither is given.

Planning tasks test the harder case, where both outputs must also be correct.