Rethinking AI Lab

Rethinking AI efficiency fundamentally

[Rethinking AI Lab @RethinkingAILab]

Excited to share our new research: vAttention - Verified Sparse Attention.
Sparse attention with provable quality guarantees for LLMs.
Full paper: https://t.co/pvOSEI8E7J
Gibhub: xAlg-ai/sparse-attention-hub
🧵 A thread 👇

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI discovers SOTA sparse attention strategies for LLM inference decoding

[ADRS Blog #12] We explore using AI to accelerate LLM inference by reducing memory traffic and latency. Using Cursor and the @skylight_org framework, we evolved a naive baseline into a sophisticated algorithm that matches SOTA sparse attention strategies for LLM decoding!⚡️

✍️ Read the Blog: https://t.co/KB49juieS0

🌐 SkyLight Website: https://t.co/na6r2lpyYl

📖 ADRS Blog Series: https://t.co/UxujLFWX8b

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

[SkyLight @skylight_org]

We’ve been assuming that extreme attention sparsity must destroy model quality.

That assumption is now false.

New SkyLight Release: vAttention ⚡️

We added vAttention to the Tier1A leaderboard. The data is striking:

👑 #1 Practical Method: Surpasses the previous lead (PQCache), closing the gap to dense models within 1% at up to 10x sparsity..
👑 Saturates Benchmarks: Dominates the sparsity–quality frontier (w/ oracle top-k).
💡 Introducing Verified-X: A new paradigm for reliable inference-time sparsity.

In this week’s blog, we share an in-depth look at the algorithm and how to run it via SkyLight 👇

[SkyLight @skylight_org]

Selecting the top-k highest scores is the intuitive standard for sparse attention. However, our latest SkyLight Tier 1A benchmarks show a non-trivial gap between Oracle Top-k and dense baselines at high sparsity.

PQCache currently tops the leaderboard as the strongest practical method, closing the gap between approximate-top-k and oracle-top-k for upto 10x sparsity levels.

Closing the gap between dense and practical sparse attention methods will clearly require new ideas. In this week’s blog, PQCache authors share an in-depth look at the algorithm, and how to run it via SkyLight 👇

[Joey Gonzalez @profjoeyg]

There has been significant recent work in sparse attention promising to unlock long context and improved efficiency -- but does it work?

We recently launched a benchmarking effort to evaluate sparse attention methods and help practitioners understand the capabilities and limitations of SoTA methods.

We are also working to make these methods more accessible in open platforms like vLLM.

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI agents generate production-ready GPU kernels that outperform compiled models

[ADRS Blog #6] This week, we feature work from @datadoghq on BitsEvolve: an ADRS framework for systems optimization. Using evolutionary search, we show how AI can automate the generation of custom, high-performance GPU kernels!

✍️ Read the blog: https://t.co/JC5TfBOyjC

🚀 Previous BitsEvolve Post: https://t.co/pUY3hc2JSS

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

💬Join the community: https://t.co/luuOMX4BIj

[Rethinking AI Lab @RethinkingAILab]

Check out SkyLight: we’re building a centralized research framework built to streamline sparse-attention development and enable rapid, grounded progress.

[SkyLight @skylight_org]

Sparse attention offers a viable path to solving the computational and memory issues of long-context LLM serving. However, we observe a surprising gap between the rapid pace of academic research and the reality of open-source adoption.

We identify the core barrier as fragmentation where new methods are published rapidly, but with missing comparative grounding.

Today we’re excited to share SkyLight, where we are building a unified framework designed to bridge this gap. SkyLight standardizes the implementation and evaluation of sparse attention to enable rigorous, controlled benchmarking.

Our first major effort is to understand the frontier of inference time sparse-decoding in LLMs.

1/N 🧵

[AI-Driven Research for Systems @ai4research_ucb]

🚀 AI optimizes tensor kernels to run 17x faster than human expert designs!

[ADRS Blog] Programming hardware accelerators is notoriously hard. We describe Autocomp, the first LLM-driven optimizer for tensor accelerators, which outperforms hand-tuned expert kernels on AWS Trainium by up to 17x!

✍️ Read the blog: https://t.co/Q9UYpben1B

📖 ADRS Blog Series: https://t.co/6cQeiIQuCq

📃 Autocomp Paper: https://t.co/syfv9spGgf

👩‍💻 Code: https://t.co/e3Bken6nrX

[Mert Cemri @mertcemri]

Building agentic systems is fun. debugging them is hell. 📉
Most failures aren’t due to weak models but due to brittle workflows.
MAST pinpoints why agents fail in an automated way, and helps you get 53% higher accuracy in your agentic system without changing the LLM. 🧵👇

[Yifan Qiao @yifandotqiao]

🚀 End the GPU Cost Crisis Today!!!

Headache with LLMs lock a whole GPU but leave capacity idle? Frustrated by your cluster's low utilization?

We launch kvcached, the first library for elastic GPU sharing across LLMs.
🔗 https://t.co/3BC7B6s2EX
🧵👇 Why it matters:

[AI-Driven Research for Systems @ai4research_ucb]

Also, our blog is available on SIGOPS!

[Rethinking AI Lab @RethinkingAILab]

This is a joint work out of Sky Computing Lab, UC Berkely @BerkeleySky @istoica05 @matei_zaharia @profjoeyg @Alex_Cuadron @luisgschroeder @randwalk0 @kumarkagrawal @Apd10Desai

[Rethinking AI Lab @RethinkingAILab]

Excited to share our new research: vAttention - Verified Sparse Attention.
Sparse attention with provable quality guarantees for LLMs.
Full paper: https://t.co/pvOSEI8E7J
Gibhub: xAlg-ai/sparse-attention-hub
🧵 A thread 👇

[Rethinking AI Lab @RethinkingAILab]

5/N
vAttention when combined with a good top-k predictor can get significantly lower approximation errors and thus model quality at small token budgets.