SkyLight

Advancing the frontier of efficient AI

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI discovers SOTA sparse attention strategies for LLM inference decoding

[ADRS Blog #12] We explore using AI to accelerate LLM inference by reducing memory traffic and latency. Using Cursor and the @skylight_org framework, we evolved a naive baseline into a sophisticated algorithm that matches SOTA sparse attention strategies for LLM decoding!⚡️

✍️ Read the Blog: https://t.co/KB49juieS0

🌐 SkyLight Website: https://t.co/na6r2lpyYl

📖 ADRS Blog Series: https://t.co/UxujLFWX8b

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

[SkyLight @skylight_org]

verified-X approach : Approximating expensive computational workflows to achieve favorable quality–efficiency tradeoffs is a promising direction. However, most approximation techniques provide little beyond empirical performance on limited or toy benchmarks, offering no formal guarantees on their behavior. This lack of guarantees is a major barrier to real-world adoption, as performance under deployment conditions remains uncertain.

vAttention addresses this gap by introducing a new class of algorithms, verified-X algorithms, that provide explicit probabilistic guarantees on the approximation error of X computation

This guarantee lays the foundation for reliable deployment of sparse attention methods in the wild.

[SkyLight @skylight_org]

We’ve been assuming that extreme attention sparsity must destroy model quality.

That assumption is now false.

New SkyLight Release: vAttention ⚡️

We added vAttention to the Tier1A leaderboard. The data is striking:

👑 #1 Practical Method: Surpasses the previous lead (PQCache), closing the gap to dense models within 1% at up to 10x sparsity..
👑 Saturates Benchmarks: Dominates the sparsity–quality frontier (w/ oracle top-k).
💡 Introducing Verified-X: A new paradigm for reliable inference-time sparsity.

In this week’s blog, we share an in-depth look at the algorithm and how to run it via SkyLight 👇

[SkyLight @skylight_org]

vAttention Computation : vAttention formalizes a stochastic approach to attention computation. Rather than relying on static sparsity patterns or fixed heuristics, the algorithm dynamically calibrates its sampling budget, treating the approximation as a rigorous estimation task.

This approach allows for provable reliability. The system provides explicit (ϵ,δ) guarantees on intermediate computations affected by the sparse approximation, ensuring that the error introduced by sparsity is mathematically bounded and controllable with high probability.

[SkyLight @skylight_org]

Food for Thought: 🤔

PQCache nearly matches Oracle performance, but here’s the catch: even Oracle methods fail to match dense attention quality.

Given the gaps on the current leaderboard, is it time to move beyond the Top-k paradigm?

[SkyLight @skylight_org]

Selecting the top-k highest scores is the intuitive standard for sparse attention. However, our latest SkyLight Tier 1A benchmarks show a non-trivial gap between Oracle Top-k and dense baselines at high sparsity.

PQCache currently tops the leaderboard as the strongest practical method, closing the gap between approximate-top-k and oracle-top-k for upto 10x sparsity levels.

Closing the gap between dense and practical sparse attention methods will clearly require new ideas. In this week’s blog, PQCache authors share an in-depth look at the algorithm, and how to run it via SkyLight 👇

[SkyLight @skylight_org]

Authors provide a system implementation in their code https://t.co/NjqeIahlw0

Key optimizations:
• Overlap CPU K-Means with GPU prefilling (no TTFT penalty)
• Cache consistently important tokens in GPU buffer
• Minimize CPU↔GPU transfer overhead

[SkyLight @skylight_org]

To learn more about SkyLight check out

👉 Blog: https://t.co/M9brzO3JWd

This is a joint collaboration from Sky Computing lab with  @apd10desai ,  @kumarkagrawal, @pd172945, @luisgschroeder  @profjoeyg ,  @matei_zaharia , @istoica05  and @RethinkingAILab

[SkyLight @skylight_org]

Sparse attention offers a viable path to solving the computational and memory issues of long-context LLM serving. However, we observe a surprising gap between the rapid pace of academic research and the reality of open-source adoption.

We identify the core barrier as fragmentation where new methods are published rapidly, but with missing comparative grounding.

Today we’re excited to share SkyLight, where we are building a unified framework designed to bridge this gap. SkyLight standardizes the implementation and evaluation of sparse attention to enable rigorous, controlled benchmarking.

Our first major effort is to understand the frontier of inference time sparse-decoding in LLMs.

1/N 🧵

[SkyLight @skylight_org]

We are open-sourcing the full Tier-1A implementation and the first standardized leaderboards today.

We invite the community to benchmark your methods against the frontier, prove your improvements on the leaderboard, and help us close the gap between sparse and dense.

👉 Leaderboard: https://t.co/RIS1V58prQ
👉 Code: https://t.co/QizTAxhQlG

5/N 🧵