Marko, soon @ ICML 2026

Meddling with the attention weights these days:
Math MSc & BSc (Jussieu / Paris Saclay)

Vegan 🌱

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@tensor_rotator A counterexample is exactly what one would expect to be found by a computer rather than a human.

Actual proofs of true conjectures (like the Erdos problems) are much more impressive.

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@tristan_j_day It just had to be done 🤣

[Marko, soon @ ICML 2026 @MarkoKarbevski]

The Multi-Head Attention satisfies the following interesting symmetry

[Tristan Day @tristan_j_day]

I’m presenting at #ICML2026 Mech Interp Workshop w/ co-author Christopher Ackerman!

We find latent #metacognition in Llama-3.1-8B: post-training/fine-tuning improve how the model uses pre-existing uncertainty and self-report structure for confidence reports and delegation.

🧵:

[ehemija.mk @EhemijaMk]

Остриот мирис на хлор што го сеќаваме во базените не доаѓа од самиот хлор, туку од соединенија наречени хлорамини.
Тие се создаваат кога хлорот додаден за дезинфекција реагира со потта, маслата од кожата и урината што завршуваат во водата од самите капачи.
https://t.co/OnGXvk77I0

[Marko, soon @ ICML 2026 @MarkoKarbevski]

And yes, this is true for the Multihead case as well. W_Q = (W_Q^1 | ... | W_Q^h) is the concatenation of the per-head weights.

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@sheriyuo Do you like the money or the freedom that they bring?

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@dviolettchan TA stuff can motivate and make you learn stuff really in depth if it's the right courses. Kind of having your brain on low engagement.

Moving apartments on the other hand...

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@dviolettchan You (only) have to find a right problem at a right time. It is partially luck.

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@GRaM_org_ Paper: https://t.co/J9KrDzbeM5 (updated version with these numbers Monday)

5/n
n=5

[Marko, soon @ ICML 2026 @MarkoKarbevski]

Should we add nonlinearity to the Query projection? Yes, if we want to spend parameters there at all.

Holding a poster at @GRaM_org_ Workshop, ICLR 2026, Sunday 26 April at 16:00, Room 101A. If you are there and this sounds interesting, please come say hi!
1/n

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@GRaM_org_ For reference, at similar scale and token budget: 12.5% larger MLP gives ~1%; EleutherAI's RoPE-vs-learned-absolute ablation: +1.78% (https://t.co/UR9LTTo4oq).

The result has come a long way: 0.5% at submission, 1.40% currently published, 2.40% in the upcoming update.

4/n

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@GRaM_org_ If linear W_Q is redundant, allocating parameters to queries requires nonlinearity. Tested it.

Q(X) = (X + f(X))/2, same parameter count. 124M GPT-3 small, 60k steps on OpenWebText:
• 2.40% lower val log-loss (6.81% lower PPX)
• Trains at 6× baseline LR

3/n

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@GRaM_org_ Under simplifying assumptions, Graef showed W_Q can be absorbed into adjacent layers (arxiv:2404.12362). Mijoski and I extended this, investigated pitfalls of propagation through LayerNorm, and provided empirical evidence that W_Q = I matches baseline (arxiv:2510.23912).

2/n

[Alek Dimitriev @tensor_rotator]

Anthropic doesn't publish often, but safety is an area where we can share knowledge to race to the top. We have an efficient two stage classifier + probes setup and I will be at our poster on Saturday at 3:15pm in pavilion 3 to answer any questions about it! Paper below 🧵

[Xiuyu Li @sheriyuo]

Had an interesting conversation in a CS PhD group chat about some humanistic topics. We discussed a TCS theory that could model the issue, and as I explained it, I had an epiphany: this could be applied to RL, a point no one has addressed before.

This is exciting because it's the first time since studying math with my supervisor that I've come up with such an idea. With his background in pure mathematics, my supervisor effortlessly generates high-quality ideas, almost like they're picked up off the street, but I can't.

Both he and another expert colleague said: this is interesting and can be done. If this works out, it would be a groundbreaking advancement in RL.

It really makes me appreciate how essential a solid mathematical foundation is for creating something both solid and innovative.

I’m truly grateful to @dviolettchan—without her, I never would have thought of this idea, even though, due to political reasons, we can't cowork directly.

[Anthony Bonato @Anthony_Bonato]

You have never peered into the heart of darkness until you have read this

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@reza_byt @icmlconf Just use the appendix

[Marko, soon @ ICML 2026 @MarkoKarbevski]

@PRooshenas @ZamAlex8 @icmlconf @grok No need to shame anyone for asking

[tokenbender @tokenbender]

implemented deepseek's mHC paper along with trainable, ablatable mHC vs HC vs base code.

at this scale (20M param, fineweb 0.9B tokens) -
- mHC has similar improvement in loss as HC
- clear mem overhead reduction in mHC over naive HC

next steps planned
- Ablations with value residuals (as asked by @Grad62304977 )
- H^res = `(1−α)*I + α*S` instead of full doubly stochastic (asked by @Nexus737326 )
- Replace sinkhorn-knopp w/ Muon's orthogonalization op (asked by @leloykun )

repo - https://t.co/ElrRrtCL1h