Yupei Du

Postdoc at Saarland University. LLM reasoning.

[Yupei Du @YupeiDu]

How do language models memorize noise while reason impressively well?

Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!

https://t.co/LD2OJmZBm5

[Nando de Freitas @NandoDF]

End to end learning — We chose no shortcuts so we could learn, and build the knowledge and infrastructure to create many more models in years to come.

[OpenAI @OpenAI]

Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946.

For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids.

An OpenAI model has now disproved that belief, discovering an entirely new family of constructions that performs better.

This marks the first time AI has autonomously solved a prominent open problem central to a field of mathematics.

[elie @eliebakouch]

Deepseek V4 Pro is the biggest open model ever with 1.6T total 49B active, trained on 33T tokens, 1M context, with 2 new attention mechanisms, Muon, mHC, open source kernels, FP4 QAT, MIT license and with one of the best tech repot of the year

[Hayden Prairie @hayden_prairie]

We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.

Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size.

Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem!

🧵👇

[Yuekun Yao @yuekun_yao]

Claude Mythos is suspected of being a Looped transformer (LT), but why are LT-based LLMs so powerful?

Our new finding: LT can perform implicit reasoning over their parametric knowledge, unlocking generalization to complex and unfamiliar questions compared to transformers ⤵️

[Michael Hahn @mhahn29]

We have 1-2 more extra spots due to new funding -- apply by end of April!

[Michael Hahn @mhahn29]

Excited about this advance in Transformer theory, giving us detailed understanding of when Transformers generalize at symbolic reasoning (and when not).

[John Schulman @johnschulman2]

Models that are great at calibrated predictions will be transformative for decision making. Excited about Mantic's work and proud they're using Tinker. Their new blog post digs into their methodology and findings.

[Yilong Chen @Yichen4NLP]

We introduce MoUE.

A new MoE paradigm boosts base-model performance by up to 1.3 points from scratch and up to 4.2 points on average, without increasing either activated parameters or total parameters.

The main idea is simple:
a sufficiently wide MoE layer with recursive reuse can be treated as a strict generalization of standard MoE.

https://t.co/UTagOXUD0y
https://t.co/LsnL5GEIaX

#MoE #LLM #MixtureOfExperts #SparseModels #ScalingLaws #Modularity #UniversalTransformers #RecursiveComputation #ContinualPretraining

[Albert Gu @_albertgu]

one question we got a lot about H-Net is how it compares to MoE. the idea is that both of them can be seen as dynamic or sparse computation methods that can adjust the FLOPs-to-parameter ratio (in H-Net, via the chunking ratio; in MoE, via number of experts). in other words for a fixed FLOP budget, both methods can increase parameter count by sparsely activating parameters only on some tokens.

in the original paper, we compared these while matching both FLOPs *and* parameters and showed that H-Net >> MoE on byte-level language modeling. of course, H-Nets can be applied to any data so an open question remained about whether it's still better than MoE when applied directly to standard tokens instead of bytes.

this paper answers the question affirmatively: H-Net seems to still consistently outperform MoE in resource-matched settings! they show this for standard language modeling (on top of BPE tokens) as well as in multimodal (vision-language models). there are a lot of other interesting results on ablations inside the architecture here

the results are cool, but the weirdest part of this paper is how hard it tried to avoid stating what they did plainly: it's literally H-Net on tokens. i think being more transparent would have helped rather than diminish the paper's results by making what they did more accessible to the community, whereas the way it is written is a bit confusing 🤷‍♂️

[Karan Dalal @karansdalal]

LLM memory is considered one of the hardest problems in AI.

All we have today are endless hacks and workarounds. But the root solution has always been right in front of us.

Next-token prediction is already an effective compressor. We don’t need a radical new architecture. The missing piece is to continue training the model at test-time, using context as training data.

Our full release of End-to-End Test-Time Training (TTT-E2E) with @NVIDIAAI, @AsteraInstitute, and @StanfordAILab is now available.

Blog: https://t.co/woCpiIrq0T
Arxiv: https://t.co/3VkFlS3wx3

This has been over a year in the making with @arnuvtandon and an incredible team.

[Shashwat Goel @ShashwatGoel7]

Excited to announce the OpenForecaster project, we train models at reasoning predict the future.

We won't get to AGI by maxxing STEM exam and coding benchmarks. That's not what most humans reason about in their day to day.

Instead, we reason about uncertainty to make decisions, using our world-model of how society evolves. Yet, there weren't any large-scale datasets to train AI for this form of reasoning. Until now.

We release OpenForesight, a training dataset of 52k forecasting questions, made from global news. Our recipe is fully automated, and can be repeated for more, newer data at low cost.

Using it, we RL trained an 8B model, and it became competitive with much larger models like GPT-OSS-120B across benchmarks and metrics.

And we want to keep building on this, in public. Our paper with full details, dataset, code etc. in 🧵

Blog: https://t.co/UhxbxmM4Te

[Songlin Yang @SonglinYang4]

the residual stream should be viewed as a recurrence, and insights from the RNN literature should apply here

[Zeyuan Allen-Zhu, Sc.D. @ZeyuanAllenZhu]

(1/N)🚀Today we launch two tightly connected milestones in the Physics of LM series: a sharpened Part 4.1 (v2.0) and a brand new Part 4.2 — together forming a clear, reproducible, textbook-style reference for principled architecture research.

Part 4.1 introduced a synthetic pretraining playground — our Galileo experiment for LLMs🍎. Our v2.0 strengthens it with Gated DeltaNet (GDN) and stricter alignment, building an even cleaner “Pisa tower” for testing architectural limits.

Part 4.2 shows these synthetic predictions resonate in reality 🌍 — across 1–8B / 1T-token pretraining — confirming which design principles actually matter.

Together, Parts 4.1 and 4.2 bring the synthetic and real worlds into surprising agreement 🤝— one more step toward a more scientific understanding of LLM architectures.

If you’re curious about:
🧠why some models reason deeper
⚙️ why linear models struggle at retrieval
🎶why a tiny horizontal mixer (Canon) changes everything …
this release ties it all together.

(Links at the end)

[Zachary Horvitz @zachary_horvitz]

✨Masked Diffusion Language Models✨ are great for reasoning, but not just for the reasons you think!

Fast parallel decoding? 🤔 Any-order decoding? 🤨

Plot twist: MDLMs offer A LOT MORE for inference and post-training! 🎢🧵

[Ronak Malde @rronak_]

My takeaways from Neurips

1. Continual learning. To support this next frontier, we’re going to need new architectures, new reward functions, new data sources, and new revenue models.

2. Neolabs. Frontier research for risky bets is being shared across multiple companies now

3. San Diego has way better weather than SF 😭

[Yupei Du @YupeiDu]

Also come visit another work of ours (poster, Nov 6, 16:30-18:00, Hall C)!

https://t.co/OLpSFTCkcI

[Yuekun Yao @yuekun_yao]

Can language models learn implicit reasoning without chain-of-thought?

Our new paper shows: Yes, LMs can learn k-hop reasoning; however, it comes at the cost of an exponential increase in training data and linear growth in model depth as k increases.

https://t.co/1CGTRJGnGQ

[Yupei Du @YupeiDu]

How do language models memorize noise while reason impressively well?

Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!

https://t.co/LD2OJmZBm5

[Yupei Du @YupeiDu]

Come talk to me (not limited to the paper!!) at EMNLP!

This is done partly during a wonderful visit to @MaiNLPlab. Many thanks to my amazing collaborators: @PMMondorf @eclipto93 @yuekun_yao Robert Litschko, @barbara_plank