Ejafa Bassam

Graduate Student @PKU1898 Researching DL/RL and applications of AI @PKUCS1978

[Aleksa Gordić (水平问题) @gordic_aleksa]

the more time i spend building these systems, the more i realize just how many research breakthroughs and compute multiplier gains are ahead of us, we're so so early

off the top of my head here are a few areas still in their infancy:

* dynamically growing model depth / extremely deep neural networks / looping
* lots of ideas in the data constrained regime ("quadrillion param" transformers, scaling weight decay, deep/shallow ensembling ; h/t to @industriaalist and the q labs team for cool recent work!)
* hyperball optimizers
* training directly on bytes / removing the tokenizer (h/t to the @cartesia team for their recent work on H-Net)

and figuring out scaling laws for all of the above (and much more)

on the systems side as well ... e.g. asynchronous multi-cluster training methods (@_arohan_ had a nice post on compute islands)

[Yusuke Miyashita @miyashita_03]

Attention sink can implicitly act as a gate. Rearranging the softmax, you get a sigmoid(this works as a gate) multiplied by the usual softmax-weighted values.

Gated attention acts as more generalised version of attention sink because it learns element-wise gating and controls each dimension independently.

Based on Qwen's paper(https://t.co/LaGhsbSxxS), there is a beautiful derivation that connects attention sink and gated attention https://t.co/D59UlfbvoM

[James Whittington @jcrwhittington]

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.

Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.

With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington

[Yingfa Chen @yingfachen]

Stronger linear attention can make hybrid LLMs weaker. Why?

This weird finding has been on my mind for a long time. We finally have a hypothesis that could explain it, and experiments that support it.

Here’s the story. 🧵
(1/5)

[Herman Brunborg @hermanbrunborg]

I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)!

The web is full of rare, repeated content that the LM can’t memorize in time. I solve this with retrieval-augmented n-gram hints🧵

[Albert Gu @_albertgu]

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

[Nathan Lambert @natolambert]

An basic idea in scaling RL: Can we allocate more compute to the harder problems?

We did this: If your GRPO group has all wrong completions, sample more with probability P (~0.9) -- in search of more GRPO batches with nonzero gradient. It works!

Called "Never Give Up"

[Ejafa Bassam @EjafaBassam]

hmm... @OpenAI cooked something good...

[Aleksa Gordić (水平问题) @gordic_aleksa]

every now and then there are research findings that reduce the quality of your sleep, and this is one of them:

[alphaXiv @askalphaxiv]

“Language Models Can Control Their Own Attention”

Long-context LLMs waste compute by scanning the entire KV cache at every decoding step, even when only a small region matters.

So this paper introduces Declarative Attention, where the model explicitly switches between global, focused, and local attention during reasoning, letting the inference engine skip irrelevant KV cache regions.

It cuts attended tokens by 52% on Gemma-4-31B and 31% on Qwen-3.6-27B with only around 1.27pp and 2.75pp accuracy drops.

https://t.co/sDUEx3Q6Ot

[alphaXiv @askalphaxiv]

“Full-bandwidth Transformer”

Transformers usually pass just the sampled token into the next step, even though the model just computed a much richer hidden state.

This paper feeds that hidden state back in too, so the model can keep working on ideas internally instead of spelling everything out token by token.

At 1B scale, it improves math, coding, and LM evals with <1% extra decoding cost, and can match models trained on ~2x more data.

https://t.co/mbAVYTdNM8

[Rosinality @rosinality]

https://t.co/RgJ4IquARQ

More scalable Muon. Instead of (XX^T)^(-1/2)X, approximating XX^T (Gram Newton-Schulz), Symmetric GEMM kernel, and communication optimization.

[marfin @marfinxx]

ByteDance and Peking University researchers just unlocked in-place test-time training by adapting MLP weights during inference

focused on LLM infrastructure engineers, inference developers, and systems leads scaling long-context AI architecture

static model weights fail on streaming long-context inputs because fixed attention layers cannot adapt to unseen distribution shifts during inference

repurposing MLP projection layers into fast test-time weights allows models to update parameters on-the-fly as context streams in

three core pillars of in-place test-time training:

→ In-Place Fast Weight Adaptation
updates final MLP projection matrices during inference without adding external side-architectures or extra memory modules

→ Task-Aligned Next-Token Loss
optimizes fast weights using self-supervised next-token prediction aligned directly with LLM pre-training objectives

→ Zero-Shot 128k Context Scaling
enables compact 4B models to process long-context streams with chunk-wise weight updates and low inference overhead

moving from static inference pass-throughs to in-place test-time training turns standard LLMs into adaptive continuous learning systems

Full paper analysis and in-place TTT benchmark evaluations below ↓

[BURKOV @burkov]

Long-context models face a tradeoff: standard Transformer attention becomes expensive as the sequence grows, while cheaper “linear attention” methods compress the past into summaries and can lose important details, especially when they merge tokens according to fixed schedules rather than what the text is actually doing.

Dynamic Linear Attention (DLA) presented in this paper from @BytedanceTalk, @OhioState, and @UMich makes that compression depend on the input: it measures how much each new token changes the current memory, keeps finer-grained states around larger changes, combines more repetitive stretches, and caps the total number of stored states by merging neighboring states with low information density, so memory use remains predictable.

The experiments across 16 reasoning, retrieval, and long-context datasets show that DLA consistently improves over the fixed-schedule Log-Linear Attention baseline, while also using less memory and achieving higher throughput.

Read with an AI tutor: https://t.co/3y4UP2zlqu

[Yoav Artzi @yoavartzi]

Almost thinking that Anthro figured out @nthngdy's backprop bottleneck for a while now, and ran with a smaller vocab for better learning...

https://t.co/fbYmiwHTxO

[Gioele Zardini @GioeleZardini]

Deep learning runs on math, but we've never had a formal language for architectures!

Now we do. Our new work is out on @TmlrOrg , led by @Vincent Abbott 🎉

https://t.co/8O1xejyA8d

[Aleksa Gordić (水平问题) @gordic_aleksa]

beautiful things are happening in mathematics, and i'm not even talking about the recent counterexample to Jacobian conjecture!

i'm talking about LLM autoformalization efforts -> we're slowly building shared mathematical infra, formalizing the foundations; this will be useful beyond maths for hardware, operating systems, dbs, GPU kernels, compilers, etc.

in this paper the authors built a harness to coordinate a large team of Claude Code agents and autoformalized a 500 page textbook in Lean

cost: $100k but they said, after identifying various orchestration/harness inefficiencies, that they can easily reduce the cost by 3-10x (without even accounting for better models)

also my good friend from OpenAI asked me to share this benchmark: https://t.co/s24aqEI9ku if you care about this space check it out!

paper: https://t.co/Ad2G0QFwOB

(great work @syhw @FabianGloeckle et al!)

[Haoyi Zhu @HaoyiZhu]

I wrote a blog on sparse linear attention: viewing sparse + linear attention through one lens — Taylor expansion — and extending it with a second-order correction (PWT) that works both training-free and
trainable.

Blog: https://t.co/jsqSfCq1fz
Code: https://t.co/RxepDYtIvO

[Xinghong (Shin) Fu @shinfxh]

Small model, looooong context

Reverso is now improved to handle significantly longer contexts, up to 16k! We pretrain our foundation model to handle inputs at multiple frequency scales, and observe an improvement in forecasting performance while maintaining fewer than 3M params

[Tilde @tilderesearch]

Introducing Online KL Shampoo (OKLS), an optimizer that brings a KL-optimal approximation of full-matrix AdaGrad to language-model training.

Diagonal optimizers ignore correlations between gradient coordinates. Full-matrix AdaGrad captures this geometry but requires quadratic state. Muon considers correlations but not their history. OKLS closes this gap using KL-optimal Kronecker factors, whitening matrix gradients across both row and column directions while remaining naturally scale-invariant.

The main challenge is computing fresh inverse-square-root preconditioners at every step. Even one-step staleness can destabilize training. We make zero-staleness preconditioning practical with Scaled CANS Coupled Newton–Schulz: 10 iterations, 27 FP16 GEMMs, and FP32 accumulation.

OKLS achieves 1.45× the parameter efficiency of Muon while retaining 98% of its training throughput. Across 200M–1B models, an OKLS model matches a Muon model roughly 1.5× larger.