Alexander Tian

[Alexander Tian @Alex2262T]

@sheggle_ Also, our paper provides a thorough analysis of why the attention mass is important to estimate for block selection. We generally show the limitations of existing methods under this analysis, which explains their lack of usage among many LLMs, and this also motivates our method.

[Alexander Tian @Alex2262T]

Hardware yearns for block sparse attention, yet it seems largely absent from open weight LLMs. DeepSeek developed NSA, and people speculated DeepSeek v4 would integrate it, yet it was never utilized.

We have a hypothesis as to why.

We found that replacing dense attention with NSA significantly degraded its ability on synthetic retrieval tasks, even when finetuned on them. On our 32k context benchmark, it scored 0.300 compared to dense attention’s 0.904.

We found the reason, and how to fix most of it. Introducing COBS🌽(Cumulant Order Block Sparse Attention), with @AdiGhai18 @sanjitneelam @ZVasania @tensorpro:

• Raises NSA’s 0.300 → 0.820, closing ~86% of the gap to dense
• 15.15x less KV-cache read traffic than dense (just 1.21x the NSA baseline)
• Lower position-wise NLL than dense in our comparison

The key insight: block selection is the keystone to block sparse attention, and existing methods are mathematically stuck in storing a first-order approximation of a cumulant generating function. COBS caches a compressed second cumulant and escapes this ceiling.

Details in the paper.

[Alexander Tian @Alex2262T]

@sheggle_ Interesting! It seems we both attempt to estimate the attention mass for block selection. HiLS uses a first order approximation and makes it learnable with some other features, while we identify the mass as a more general CGF, of which we use the first two terms in COBS.

[Alexander Tian @Alex2262T]

Paper: https://t.co/mN8iRC1dk1