Hengyu Fu

PhD student of Computer Sciences at UC Berkeley

[Hengyu Fu @HengyuF]

Frontier labs are rumored to be adopting looped Transformers for their potential to unlock implicit reasoning. But can looped Transformers really master it?

We find a bottleneck that keeps looped Transformers from perfect implicit reasoning. Using mechanistic interpretability, we show how looping helps implicit reasoning, why it still struggles to generalize, and how to fix it.

We introduce DiscoLoop: a residual design for looped Transformers that improves implicit reasoning.

🧵 Long thread ahead 👇

[Hengyu Fu @HengyuF]

Please check out our new NeurIPS 2026 paper! 🎉 We find why looped Transformers still struggle with implicit reasoning, and fix it with DiscoLoop.

[Hengyu Fu @HengyuF]

Frontier labs are rumored to be adopting looped Transformers for their potential to unlock implicit reasoning. But can looped Transformers really master it?

We find a bottleneck that keeps looped Transformers from perfect implicit reasoning. Using mechanistic interpretability, we show how looping helps implicit reasoning, why it still struggles to generalize, and how to fix it.

We introduce DiscoLoop: a residual design for looped Transformers that improves implicit reasoning.

🧵 Long thread ahead 👇

[Hengyu Fu @HengyuF]

Thank you! These are really good questions.

We did try feeding only the decoded embeddings into the next loop during training, but optimization broke down and the model did not generalize well. Our hypothesis is that keeping the continuous state mainly helps optimization and carries contextual information from the input (e.g., the relation). The continuous channel leaves a smooth path through the recurrence, so gradients can flow across loops without being routed entirely through the decode-then-encode operator.

We haven't tried changing the size ratio between graph A and graph B (or adding more A-like graphs) yet, but it's definitely worth trying!

[Hengyu Fu @HengyuF]

Frontier labs are rumored to be adopting looped Transformers for their potential to unlock implicit reasoning. But can looped Transformers really master it?

We find a bottleneck that keeps looped Transformers from perfect implicit reasoning. Using mechanistic interpretability, we show how looping helps implicit reasoning, why it still struggles to generalize, and how to fix it.

We introduce DiscoLoop: a residual design for looped Transformers that improves implicit reasoning.

🧵 Long thread ahead 👇

[Hengyu Fu @HengyuF]

@M_Decoherence Yes please see our paper at https://t.co/gKRcK4WLhZ

[Hengyu Fu @HengyuF]

@Skywalkerlyzv Thank you so much🫡

[Hengyu Fu @HengyuF]

@biosemiote Thank you! Yes, this reinjection operation is the key to unlocking OOD generalization in the multi-hop tasks!

[Hengyu Fu @HengyuF]

@davidyang05 Just coincidence 🤣 Our paper was published in July before the startup.

[Hengyu Fu @HengyuF]

@SEI_North Thank you so much! For me details, please check our paper at https://t.co/gKRcK4WLhZ

[Hengyu Fu @HengyuF]

@SreeramG Thanks! It's a soft, learned mix rather than a hard swap from hidden states to embeddings. The design principle is that later loops should get inputs the model knows how to read. Better ways to close the mismatch are very open!

[Hengyu Fu @HengyuF]

@RayLu_THU Thanks Rui! Your advice on the manim animation is super helpful!

[Hengyu Fu @HengyuF]

For simplicity we used a fixed number of loops during training. But Discoloop is completely compatible with adaptive looping! In multi-hop tasks, we find that looping more than the required reasoning depth can even refine the accuracy!

Figure below shows a looped model trained on multi-hop questions with at most 5 hops. At test time, looping more than 5 times raises the OOD accuracy significantly.

[Hengyu Fu @HengyuF]

@shuizhuyu Thank you so much! We summarized the initial pretraining results in [5/6] of our thread and more details in the paper!

[Hengyu Fu @HengyuF]

Thanks for your advice! We chose full layer loop to ablate the “local-depth storage” issue in the multi-hop reasoning tasks, but it is completely worth trying middle layer loop! This also reminds me of a fantastic blog comparing full layer loop and middle layer loop in real pretraining https://t.co/DCxvYsNs8S

[Hengyu Fu @HengyuF]

@generatorman_ai Agree! Decoding/sampling serves as a realignment back to the input space.

[Hengyu Fu @HengyuF]

@arundsharma @latentpedia Thank you! This is a very interesting perspective! For knowledge retrieval, it makes perfect sense to use outer structures system like RAG and build a graph for direct retrieval of the multihop answer!

[Hengyu Fu @HengyuF]

@KevLXu Complete coincidence 🤣Our Discoloop paper was published in July, prior to the startup's launch. The two are entirely unrelated.

[Hengyu Fu @HengyuF]

Thank you! That's exactly why mechanistic interpretability matters: it lets us read an LLM's hidden thoughts. Looping is interesting here too. In real language modeling, it may not just add reasoning depth. It may also refine the looped tokens as they settle toward a fixed point.

[Hengyu Fu @HengyuF]

@LiangTwistShan Thanks a lot! Hope it brings some value!

[Hengyu Fu @HengyuF]

@_Yann77 Thanks a lot! Glad you found it helpful!

[Hengyu Fu @HengyuF]

@anantg4rg Thank you! We believe looping is a new axis to scale up compute in a parameter/memory-efficient way!

[Hengyu Fu @HengyuF]

@MH2023ML @RayLu_THU 🤣