Ferdinand Kapl

PhD Student @TU_Muenchen @helmholtz_ai

[Mihir Mahajan @maharajamihir]

Today we release crowd-code 2.0, our second phase of crowd-sourcing the largest long-horizon open software engineering dataset.

Install once. Forget about it.

[Ferdinand Kapl @KaplFer]

📜 Do Depth-Grown Models Overcome the Curse of Depth?
An In-Depth Analysis

Paper: https://t.co/bAiHrIhR4k

[Ferdinand Kapl @KaplFer]

SOTA LLMs are trained as large monolithic models. But looping or depth growth can beat that. How?

Co-led with @AngelisManos and @TobiasHppe1.

Incredible collaboration with Paradigms of Intelligence Team @Google 🙌
@kaitlinmaile
@oswaldjoh
@ninoscherrer
and @stefanAbauer. 🧵🔽

[Ferdinand Kapl @KaplFer]

🔮 Outlook: towards Looped Transformers.

With depth growth, we see compact computational blocks repeated along depth with an internal cyclical structure. This is reminiscent of Universal Transformers: loop-like computation emerges from depth growth without explicit weight tying.