Marko Šišović

swe @databricks by day
pretraining research by night

[Marko Šišović @sisovicm]

one intuition behind this is that linear attn can act as a learned, data-dependent positional bias. in fact they conclude that rope is not only redundant, but hurts the performance

paper: kimi linear (https://t.co/i6RdQLEsTU)

[Marko Šišović @sisovicm]

previously i thought linear attention was about trading off some long-context performance for major efficiency/computational gains. turns out a mix of linear attn and mla can outperform pure mla on long context

[Marko Šišović @sisovicm]

@gordic_aleksa yes, that was interesting to me as well. kimi linear reports the same

[Marko Šišović @sisovicm]

@vvvincent_c Very nice breakdown! I was too skeptical when I forst saw the PR, as it didn't even include logs at the time. Btw, the "does it scale" part is a great idea.

[Marko Šišović @sisovicm]

@m_sirovatka Always thought design patterns were such a curse 💀

[Marko Šišović @sisovicm]

@Grad62304977 It can still be dynamic between conversations though? Like effort level set on the start of the conversation. Still much less flexible than CoT, but probably worth noting.

[Marko Šišović @sisovicm]

@Creative_Math_ Pretty basic rec, but it's still rly good, covers all the basics: https://t.co/z6AJKIK4iV

[Marko Šišović @sisovicm]

@benbye @lucasmaes_ This looks super dope!

[Marko Šišović @sisovicm]

@ethanmclark1 Not true that representation prediction is the only one doing self-supervised pretraining and then SFT on top. Cosmos uses the exact same recipe on a high level.

[Larry Dial @classiclarryd]

New NanoGPT Speedrun WR at 79.2 (-0.5s) from @sisovicm, by moving the MLP up projection to FP8 on the forward pass, and increasing step count by 5 to offset the loss increase. The march towards lower precision continues! https://t.co/HSaA2PR6gv

[Marko Šišović @sisovicm]

@classiclarryd @yzhang_cs Wow didn't expect such speedups to be possible still! Looks like residual mixing keeps on giving good results, and is maybe even still underexplored.

[Marko Šišović @sisovicm]

@iamgrigorev Very cool! Why were you hesitant, and how are the loss curves looking compared to mHC, or do you not have clean comparisons since mHC proved too complex.

[Marko Šišović @sisovicm]

@eliebakouch Combined with the fact that post-train perf is downstream of pre-train, makes you wonder if they are really using compute optimally.

[Marko Šišović @sisovicm]

@eliebakouch It's interesting to compare to Kimi K2.6, also a post train extension of K2.5: Terminal-Bench 2.0 66.7 and SWE-Bench Multilingual 76.7. While the scores are better they aren't dramatically so.

[Marko Šišović @sisovicm]

@Dorialexander Ah yes, I overlooked that part. So you could force this on the batch level, similar to how you force the load balancing?

[Marko Šišović @sisovicm]

@Dorialexander Although to be fair, dense MLP could also be viewed as having "soft" routing, so I could also see this MoE effect being smaller than expected.

[Marko Šišović @sisovicm]

@Dorialexander Very interesting, so it could be that MoE is more robust to variance in the training distribution because of training dynamics.

The router groups similar tokens, and thus similar sequences, to the same experts, so anomalous modes are localized to just a few experts.

[Marko Šišović @sisovicm]

@andrewbriand8 wow, didn't even know kernels could be instruction issue limited, very cool