🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 https://t.co/DpVhu2b44F
due to (no so) high demand, I make a vanilla version of Angry Birds Beats Reviewers.
(The DepthBench & Angry Birds dream collaboration animation is in cooking, stay tuned.)
@khtsly_ Interesting, thx for sharing! Will take a deeper look👀
btw my colleague proposes Linear Depth Memory which shares quite relevant intuition (https://t.co/7212YnY97W) and prefers relevantly large aspect ratios
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 https://t.co/DpVhu2b44F