I think we are missing the nonlinear nature of human intelligence when interacting with these LLMs.
I am not talking about activation functions. I am talking about fundamental system, which is a linear chat. I don’t think it’s a great way to interact with LLMs.
I want to have something which lets me compose and decompose different parts of the chat as per the requirements. Think of human brains: you just don’t keep dumping information in a linear way, we choose information which is relevant at a time, we don’t do compression, we don’t keep building context. we do fundamentally something different.
@thsottiaux Devise a two-dimensional interface for agentic coding. I doubt that a linear chat suits coding tasks well.
One approach to this sort of issue is to make each turn pluggable. We should be able to add or remove different parts of turns as needed.
Zyphra Hierarchical Attention (ZHA) follows a similar direction, but differs in how it groups and retrieves KV entries. While LSA uses semantic clustering and top-k/beam search, ZHA uses contiguous groups, learned summaries, and adaptive top-p routing across multiple hierarchy levels. Both aim to make long-context attention subquadratic by reducing retrieval overhead.
Spent considerable time dissecting Latent Sparse Attention (LSA), expecting a fundamentally new attention mechanism.
Turns out, the core idea is surprisingly simple.
Compress for retrieval. Expand for attention.
Here's how it works:
Project & Pool: Project keys into a lower-dimensional indexing space and sum consecutive key vectors into compact representatives. The original KV cache remains untouched.
Cluster: Group pooled representatives by semantic similarity, even if they originate from distant token positions. Store centroids and mappings to original KV positions.
Search: Compare queries against cluster centroids. Tree LSA uses hierarchical search to avoid scanning every cluster.
Expand: Select the most relevant clusters and retrieve their original, uncompressed KV entries using stored indices.
Attend: Combine retrieved entries with recent-window and sink tokens, then perform standard softmax attention.
Two similarity computations, but only one actual attention operation.
The real contribution is computational complexity.
Conventional sparse attention can still require (O(T^2)) indexing work during prefill. Tree LSA reduces attention/indexing complexity to (O(T\log T+Tk)), subquadratic for a fixed retrieval budget (k).
The paper reports 22.7× faster attention-layer prefill at 10M tokens.
What looked like an elephant turned out to be a mouse conceptually. But engineering that mouse to scale is the real achievement.
The breakthrough isn't making attention sparse. It's making sparsity cheap to discover. Is there anything I missed
Paper: https://t.co/vABoWkmXVz
#LLMInference #MLSystems #AIResearch
@rikiyatakehi@a1zhang@lateinteraction@jayden_teoh_ I have a refined approach to understanding why a larger model alters the weights of a smaller one. I believe this process is known as gradient descent, have you encountered that term?
There’s only one way Starlink can operate in India: if Starlink becomes an independent entity with no government influence over them in any situation, not even in geopolitical conflicts, and I don’t think that’s possible for Starlink.
Either transfer control of Starlink Infrastructure to the Indian Telecom Partners so they can operate it and receive royalties, or keep crying!
@Kanthaswam80359 There are some areas in Indian which are sensitive and if those areas got access to Internet, bad actors can use for malicious reasons