Ryan Lee

ML/AI researcher

[Dhananjay Ashok @Dhananjay99Ash]

Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games:

Paper: https://t.co/05Fpom1OM1
Website: https://t.co/qFUZfld80b

[Ryan Lee @_ryantlee]

@ahatamiz1 Great research directions! On it 🫡

[Ryan Lee @_ryantlee]

@reza_byt @ahatamiz1 Your critique of real vs theoretical flops savings is real though: I definitely want to address that in the future and totally agree that MoR has more thoroughness on how to accomplish it. Just a bit compute constrained as a university researcher.

[Ryan Lee @_ryantlee]

@reza_byt @ahatamiz1 Hey Reza, great points: I’ll take a closer read of MoR so that your work is better represented in the camera ready. Happy to discuss follow-up studies. Lmk, but I think the delta of our paper is: just MoE+plain loops improves looped LM scaling, natural (untrained) vs learned EE.

[Ryan Lee @_ryantlee]

@waxhn @jbiloki @edwardjhu @jonathanmay Max, i was reading through your post, very cool: do you think a solution could be to pretrain at mixed depths (and KVs) vs fixed depth to improve EE generally? Obv training from scratch full scale Huginn would be too much, but we can test on small(er) scales like i did.

[Ryan Lee @_ryantlee]

My first PhD paper, 'Sparse Layers are Critical to Scaling Looped Language Models,' is accepted at NeurIPS 2026! Looping & early exits enables adaptive depth; MoE makes it scale. First step toward compute-adaptive LMs.

w/ @jbiloki @edwardjhu @jonathanmay
https://t.co/bjxI5K1siX

[Ryan Lee @_ryantlee]

@AfshinK91 @ahatamiz1 I agree, and we definitely talk about FLOPs (active vs. stored) in our paper, see section 3.2 and Table 3! its actually a very important distinction as you rightly pointed out, esp. when throwing MoE in the mix.

[Ryan Lee @_ryantlee]

@waxhn @jbiloki @edwardjhu @jonathanmay Hmm, thats an interesting finding! Thanks for sharing, would love to learn more. While I agree that a perfect KV is an idealized assumption, i think the penalty applies equally to our baselines too: which would also need to use whatever prior KV at the layer they exited prior.

[Ryan Lee @_ryantlee]

@waxhn @jbiloki @edwardjhu @jonathanmay Hey Max: the early exit analysis here is done with teacher forcing, i.e. tokens up to token i are perfect and we look at the i’th token’s entropy at layer exits. There are more recent papers that tackle loop KV entries, to grossly summarize them, you copy the last available KVs.

[Ryan Lee @_ryantlee]

@henrytdowling @jbiloki @edwardjhu @jonathanmay Great Q: MoEs still beat out Looped-MoE. It's not that MoE completely solves the looping penalty, but that Looped-MoE is probably the best looped model we can build rn. Early exits with Looped-MoE are certainly better than with MoE, which is what I'm more excited about actually.

[Ryan Lee @_ryantlee]

@wangsw5653 Nice work, thanks for the cite in section 2! Would love a row in Table 1: we match (active) params for Looped-MoE architectures against baselines, so forward FLOPs match, and effective depth is fixed at 16 (8x2 vs a 16-layer Base), so KV matches too. https://t.co/bjxI5K1siX

[Edward Hu @edwardjhu]

How does one RL post-train a 397B model for long-horizon knowledge work? 👩‍💼

We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research.

Full blog: https://t.co/GWgR2DB7iy
Source code: https://t.co/96h8ADfvBp

[Ryan Lee @_ryantlee]

@wangsw5653 Nice work, thanks for the cite in section 2! Would love a row in Table 1: we match (active) params for Looped-MoE architectures against baselines, so forward FLOPs match, and effective depth is fixed at 16 (8x2 vs a 16-layer Base), so KV matches too. https://t.co/bjxI5K1siX

[Ryan Lee @_ryantlee]

@MrCatid @rosinality Agree 100% but MoE does require you to store more parameters than standard dense models. Looping is nice because you can balance storing unused sparse weights with re-using layers -> you get a compute optimal model that doesn’t break memory bank and performs well

[Ryan Lee @_ryantlee]

@rosinality Thanks for the share!

[Ryan Lee @_ryantlee]

@ChangJonathanC @rosinality @nopainkiller In fact, we observe that a majority of tokens at the same layers get routed to a unique set of experts when looped, even without specific training, just standard expert balancing losses. (Section 6.1, figure 5). I’ve considered encouraging reserving experts for layer, though!

[Xuezhe Ma (Max)✈️COLM @MaxMa1987]

After about 2 years, we are proud to release Gecko, an efficient architecture that improves upon Megalodon, with capability of efficiently and inherently processing sequences with unlimited context length.

One of the most important idea in Gecko is Adaptive Working Memory(AWM), implemented using a linear attention mechanism with a position-aware online softmax activation. Notably, AWM globally compresses information into memory, rather than discarding historical information through forgetting.

In a controlled head-to-head comparison with Llama2 and Megalodon, Gecko achieves better performance in the scale of 7B and 2T training tokens. Gecko achieves 1.68 training loss, vs. 1.67 of Llama2-13B, with half number of parameters on 2T tokens.

Paper: https://t.co/hLJZ9VPnea
Code: https://t.co/UPNhjlvNq3