Jiahan Li

PhD @NYU_Courant w/ @rob_fergus & @kchonyc, SR @GoogleResearch, Previous: @pku1898 Working on Generative Models, AI4S

[Jiahan Li @JiahanCed]

Why do most unified models still stitch together two different paradigms: an autoregressive LLM for text, and an attached diffusion head for images? (1/16)

We wanted to see if one generative process could handle both modalities on equal footing.

We introduce JUICE 🧃 (Joint Unified multimodal modeling In Continuous Encoder spaces): a flow-matching Transformer (MM-RiT) trained from scratch over frozen pretrained image and text encoders, with no LLM initialization.

Each 2.1B model uses one set of weights:
- SigLIP2 variant: 0.87 GenEval, 85.27 DPG-Bench, 0.45 WISE, 73.1 VQAv2
- DINO variant: 0.88 GenEval (tying TUNA, the strongest 1–3B baseline)
- Parallel text decoder: 3.9–6.4× faster than AR on VQA, 54–163× on captions

Blog: https://t.co/Kq3m5QksMH
Paper: https://t.co/9E5LWVJCKA
Weights: https://t.co/iIM9FjtJ8P
Code: https://t.co/efBAPow42j

[Zizhang Li @zizhang_li]

🌍 Actions that condition world models are highly heterogeneous: forces on objects, camera movements, hand motions, robot arms, each with its own meaning and structure. So today's action-conditioned world models are usually built one action type at a time, each with its own conditioning design.

We introduce UniAction, a unified action-conditioning framework: one world model driven by all of these actions, and even by mixes of them in a single generation. Trained jointly, it outperforms the action-specific experts. Great work led by amazing @BowenXue_ .

More detailed threads below👇
🧵 1/10

[Jiahan Li @JiahanCed]

🔥🔥🔥

[Kyunghyun Cho @kchonyc]

Ortet's mission is to help create healthcare where everyone can receive predictably better care. in pursuing this mission, we believe predictability must come together with transparency and controllability.

taking the first step in this direction of transparency and predictability in health, we have created the Orion cluster, host a large fleet of open-weight models and began our evalution campaign on health-focused benchmarks.

in doing so, we learned a lot about building up an agent-first cluster and also learned about the current state of AI models in the context of health benchmarks. we are excited to share the first set of findings from this ongoing campaign on our Orion cluster.

[Morgan Levine @DrMorganLevine]

I've been meaning to share these thoughts for a while and touched on them briefly in a recent conference talk. They are relevant to a lot of the discussions today. https://t.co/YywkAS87eQ

[Diego del Alamo @DdelAlamo]

@RolandDunbrack My current mental model of ipSAE is that the low end (depends on the target) is extremely predictive of nonbinding, but higher values are not. So it is useful but not perfect at filtering. Per @amirshanehsaz's comment, 11/60 ain't bad!

[Jimmy Heaters @CathPoaster]

If you think San Francisco is only 4-5 years behind NYC culturally then you are a philistine idiot. San Francisco is decades behind NYC and anyone with an iota of taste in their body knows this. NYC clears SF in food, night life, music, arts, fashion, urbanism, walkability, intellectual spaces, and exposed brick walls. If you have observed how people in the two cities even just dress and carry themselves you know it's a hydrogen bomb vs coughing baby situation. NYC has comedy clubs, theater, and intellectual life spanning EVERYTHING, not just AI and tech. In New York I can go see a play, get dinner in a dimly lit restaurant, and then get drinks with my friends for like $500. You can't do that in SF. In NYC I can go to a party in Brooklyn and meet amazing experimental visual artists who really don't like getting asked if their parents pay for their nice 1 bedrooms. Yesterday I went bar hopping during the day to catch all the Big 10 football games which is becoming a huge cultural mainstay in Manhattan. Yeah, we're culturally and intellectually superior to San Francisco tech nerds. Do I even need to mention the subway?

[Jiahan Li @JiahanCed]

@Pandatree537201 Thanks for suggestions! I'm curious why your text has so many em dashes🥹😂

[Jiahan Li @JiahanCed]

Why do most unified models still stitch together two different paradigms: an autoregressive LLM for text, and an attached diffusion head for images? (1/16)

We wanted to see if one generative process could handle both modalities on equal footing.

We introduce JUICE 🧃 (Joint Unified multimodal modeling In Continuous Encoder spaces): a flow-matching Transformer (MM-RiT) trained from scratch over frozen pretrained image and text encoders, with no LLM initialization.

Each 2.1B model uses one set of weights:
- SigLIP2 variant: 0.87 GenEval, 85.27 DPG-Bench, 0.45 WISE, 73.1 VQAv2
- DINO variant: 0.88 GenEval (tying TUNA, the strongest 1–3B baseline)
- Parallel text decoder: 3.9–6.4× faster than AR on VQA, 54–163× on captions

Blog: https://t.co/Kq3m5QksMH
Paper: https://t.co/9E5LWVJCKA
Weights: https://t.co/iIM9FjtJ8P
Code: https://t.co/efBAPow42j

[Jiahan Li @JiahanCed]

@JinWeiyang18434 Thanks sharing! Well I still think we need someway to inject world knowledge from I’ll to these generative models / Or maybe adding heavy uni-modal language data.

[News from Google @NewsFromGoogle]

Gemini 4 Argon is already helping researchers unlock new scientific breakthroughs across fusion, biology, and quantum computing:

⚛️ In nuclear fusion, Argon helped discover a broader class of counterexamples to Grad's conjecture — a nearly 60-year-old assumption in plasma physics — creating new benchmarks for stellarator 3D magnetic fields.

🧬 In biology, it designed five novel, lab-verified serine esterases in a matter of hours — enzyme engineering that would previously have taken months. This work demonstrates the potential for rapid acceleration in fields such as biomanufacturing, green chemistry, and drug development.

🩺 In rare disease research, scientists at @GamowLabs used Argon to analyze de-identified genomes from newborns with rare developmental lung disorders — pinpointing a chromosomal rearrangement that routine software missed for years, and explaining why it went undetected.

🧮 In quantum computing, Argon translated our AlphaQubit error-correction decoder into a single optimized hardware kernel — nearly doubling inference throughput — and cut spacetime costs for a key quantum subroutine by more than 100x.

[Jiahan Li @JiahanCed]

@swe_acc lol we should shout out to Unidiffuser https://t.co/C6x6puSQdG

[Jiahan Li @JiahanCed]

@owengretzinger Thanks!

[Jiahan Li @JiahanCed]

Thanks ! We empirically use the same distribution for images and text; for example, for images we use the distribution from the RAE paper.
I think we could sweep time distributions to find a better config, but the original distributions already work very well (and widely used in different papers lol).

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

What if multimodal generation didn’t have to choose between understanding and creating, or between images and text going first?

Excited to share JUICE 🧃 with @JiahanCed and the team!

What I find especially interesting is the role of representations: the same features that the model uses to understand an input also become the states it learns to generate. Images and text can then take shape together, with each modality providing context for the other throughout generation.

One detail from our ablations: even when only one modality is supervised on joint examples, keeping the noisy partner stream improves image generation over conditional-only training at the same target count. Cross-modal interaction can be useful before either side is fully formed.

The maze experiments make this tangible. A drawn route and a written sequence of moves can each be valid while describing different solutions. Coupled generation lets the two streams settle on a consistent pair, without choosing in advance which modality goes first.

This also raises an interesting question for multimodal memory: could similar interactions help models recover useful context from incomplete visual and linguistic cues?

Excited to explore where this direction leads!

[Jiahan Li @JiahanCed]

@kartikb753 Totally agree! Maybe we need better language or multi-modal encoder? Or we need to add more uni-modal language data during training.

[Jiahan Li @JiahanCed]

@RyanJ_KOL Yes, we include some results in the paper's appendix.
Honestly, I still feel there's a long way to go for continuous diffusion language models; in this paper, we show that modeling language in continuous space helps a lot with multimodal tasks.

[Jiahan Li @JiahanCed]

Thanks @Huangyu58589918 @TongPetersb @shushengyang for endless discussions and support, @RuihanGuo2 @hungsuzh143318 @zhaohexu2001 for fun ideas and advice, @pratyusha_PS @sherryyangML @thoma_gu for guidance, and above all my advisors @rob_fergus @kchonyc for always backing me.

[Jiahan Li @JiahanCed]

Results 4/5: joint reasoning on mazes.

On synthetic mazes with 2–16 shortest routes, our task-adapted 2.1B SigLIP2 model (224×224) draws a route and writes its moves; both must be valid and match (93.7% coupled vs 54.6% independent):
- Independent sampling: 54.6%, as each stream often picks a different valid route
- Text first / image first: 85.3% / 95.2%. The more reliable modality should go first, but on a new task we don't know which one that is in advance.
- Coupled sampling (both streams updated together): 93.7%, within 1.5 points of the better fixed order without choosing one

Generation order is no longer hardwired: it becomes a sampling-time choice, and coupled sampling can leave it open. In the two traced mazes, whichever output settles later adapts to match the other.

[Jiahan Li @JiahanCed]

Results 5/5: planning by drawing and writing.

On visual spatial planning (VSP, FrozenLake), our task-adapted 2.1B SigLIP2 model (224×224, trained on synthetic maps) draws the path and writes the moves together with coupled sampling:
- Training grid sizes (3×3 to 6×6): 99.8% success, on par with LatentUM-fine (99%). LatentUM-coarse reaches 85%, ThinkMorph 76% and InternVL3.5-38B 20%.
- Unseen larger grids: 94% on 7×7 and 69% on 8×8.

On unseen sizes, the drawn path is more reliable than the text moves, matching the wording cost from Takeaway 1. On 9 of the 600 benchmark maps, the moves initially fail and are corrected during coupled sampling; in the 8×8 map shown, they first walk into a hole.

[Jiahan Li @JiahanCed]

Results 3/5: an image and a caption from noise, together.

With no condition, both streams start from noise and we generate an image and a caption at once. Because the text clock runs ahead, the caption takes shape before the image does, while each stream conditions on the other at every step. In the selected samples, each caption describes the image generated alongside it, even though neither is given.

Planning tasks test the harder case, where both outputs must also be correct.

[Jiahan Li @JiahanCed]

Results 2/5: visual understanding, same weights.

With the image and question held clean, the same SigLIP2 model generates the answer's text states and reads them out in one parallel pass: 73.1 VQAv2, 58.6 GQA, 66.0 MMBench, and 84.0 POPE (224×224, no LLM initialization).

Where training without an LLM falls short is domain-knowledge QA: on AI2D (34.4) and MMMU (29.6), SigLIP2 trails LLM-initialized 1–3B models (58.5–71.4 on AI2D). Pretrained encoders give strong visual grounding, while domain knowledge is where LLM initialization still pays off.

[Jiahan Li @JiahanCed]

Results 1/5: text-to-image at 2.1B.

- DINO: 0.88 GenEval, tying TUNA 1.5B, the strongest 1–3B baseline
- SigLIP2: 0.87 GenEval, 85.27 DPG-Bench (between Show-o2's 85.02 and TUNA's 86.03) and 0.45 WISE, the highest among 1–3B unified models that report it

With this recipe, we trained two 2.1B models from scratch, one per image encoder (224×224, no LLM initialization): 60k pretraining updates, then 15k finetuning updates on 20.6M text-to-image and understanding examples. Each model uses one set of weights for text-to-image, VQA and joint generation.

[Jiahan Li @JiahanCed]

Pretraining, in short: four controlled comparisons, four takeaways.

1. Continuous text spares images but costs wording.
2. Train where you sample: the edges and the interior.
3. Semantics over reconstruction.
4. Scale reorders image spaces for generation, not captioning.

Together they fix the recipe for the 2.1B models: continuous text states, joint plus conditional training, and semantic encoder spaces (SigLIP2 or DINO for images, Flan-T5 for text).