Jack

In the land of the blind the one-eyed man is king

[Jack @JackNotOld]

Put out my first LessWrong blog post!

Interpretability treats steering directions like "control knobs". I checked whether that assumption is mathematically valid across 8 different models.

At α = 1, it breaks in 92% of cases.

https://t.co/esn7gJZ0wM

[Jack @JackNotOld]

The @ElevenLabs robot is serving coffee in SoHo.

[elie @eliebakouch]

WOW microsoft new "MAI Thinking 1" model comes with a 109 page tech report that looks REALLY detailed, this is amazing

[Jack @JackNotOld]

@Andres_Nava_12 @MatthieuWyart Awesome!

[Jack @JackNotOld]

@corefpark Yeah for sure am curious whether atlantis lands in a coherent off-manifold structure after distance
fine-tunes or just scatters. let me know if you run it.

[Ali Hatamizadeh @ahatamiz1]

Gated DeltaNet-2 is here. 🚀

🔥 New paper: Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Gated DeltaNet-2 outperforms KDA and Mamba-3, the latest and best recurrent architectures, head to head at 1.3B. 🏆

💡 Here's the idea behind it:

Linear attention squeezes an unbounded KV cache into a fixed-size recurrent state. The hard part isn't just what to forget, it's how to edit that memory without scrambling the associations already in it.

Prior delta-rule models like Gated DeltaNet and KDA use one scalar gate to do two jobs at once: erasing old content and writing new content. But these two decisions act on different axes of the state, so tying them together is a real limitation.

Gated DeltaNet-2 decouples them.

✂️ a channel-wise erase gate b_t picks which key-side coordinates to read and remove
✍️ a channel-wise write gate w_t picks which value-side coordinates to commit
🔁 recovers KDA when both gates collapse to a scalar, and Gated DeltaNet when the decay collapses too
⚡ still trains fast: chunkwise WY algorithm with gate-aware backward, fused in Triton

📊 Results:

We train 1.3B models on 100B tokens of FineWeb-Edu, matched in recurrent state size, against Mamba-2, Gated DeltaNet, KDA, and Mamba-3.

Best average on language modeling + commonsense reasoning, in both recurrent and hybrid settings
Biggest gains on long-context RULER retrieval. S-NIAH-3 jumps from 63 to 90 over KDA, and multi-key needle retrieval climbs from 28 to 38

Joint work with @YejinChoinka and @jankautz.

📄 Paper: https://t.co/Zw6yXbHjGU
💻 Code: https://t.co/s8IWwaRU18

#LinearAttention #StateSpaceModels #Mamba #LLM

[Jack @JackNotOld]

@eliebakouch Crazy timing!

https://t.co/FjFuPDJiru

Just published paper about SAE features in Qwen recurrent writes that behaves like an erase operation

Feels like these architectures are converging toward increasingly interpretable state dynamics

[Jack @JackNotOld]

New paper! Trained an SAE on Qwen's recurrent state writes.

Found an "erase" feature. Substituting it for the model's "write" drops the target token from next-token logits. The shift factors through forget, read, output at R²=0.98 with no fitted params.

https://t.co/XrlF3Ekx9G

[Jack @JackNotOld]

Interesting, though gradient misalignment alone doesn’t necessarily mean a separate manifold... distance task may induce a uniquely structured gradient direction (making it look orthogonal to other tasks) even with same underlying geometry

Would be really interesting to see whether those gradients project onto the same principal directions as the other tasks

[Jack @JackNotOld]

HF: https://t.co/oh8DRVK8RU
Code: https://t.co/cSXpZMo2AR

[Jack @JackNotOld]

New paper! Trained an SAE on Qwen's recurrent state writes.

Found an "erase" feature. Substituting it for the model's "write" drops the target token from next-token logits. The shift factors through forget, read, output at R²=0.98 with no fitted params.

https://t.co/XrlF3Ekx9G

[Jediah Katz @jediahkatz]

i would never hire anyone with a 4 year resume gap

[Citrini @citrini]

Morgan Stanley’s price discovery happens on @tradexyz

[Jack @JackNotOld]

“Directionally very interactive”

[Thinking Machines @thinkymachines]

People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way.

We share our approach, early results, and a quick look at our model in action.

https://t.co/AFJZ5kH7Ku

[Jack @JackNotOld]

@natolambert @scaling01 Congrats!

[Jack @JackNotOld]

@edwardjhu @OpenAI @Yoshua_Bengio @Mila_Quebec Congratulations!

[Jack @JackNotOld]

Trained states and dataset now on @huggingface Hub 🤗

Hybrid models (Qwen3.5, FalconH1) initialize 75% of their parameters to zero. We trained those initial states on 45 verified solutions: +23.6pp on HumanEval, +10.8pp over LoRA, zero inference overhead.

Try S₀ tuning on Qwen3.5-4B without training:
https://t.co/EyLKeEkveI

Training data (45 verified HumanEval solutions):
https://t.co/gyXwOnyq74

Github: https://t.co/A4KwMeGdYA
Paper: https://t.co/fsakFtE5Vj

[martin_casado @martin_casado]

Mythos appears to be the first class of models trained at scale on Blackwells. Then will be Vera Rubins. Pre-training isn't saturated. RL works. And there is *so much* computing coming online soon.

Buckle your chin strips. It's going to be fucking wild.

[Jack @JackNotOld]

@samsheffer @GoogleDeepMind @OfficialLoganK @GoogleAIStudio Congrats!

[Jack @JackNotOld]

Code: https://t.co/MqrEhJs4hR
Paper: https://t.co/KeFrHyzBhQ
pip install s0-tuning

This suggests a different axis of adaptation: state, not weights.