Aditya Desai

Post Doc, UC Berkeley

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI discovers SOTA sparse attention strategies for LLM inference decoding

[ADRS Blog #12] We explore using AI to accelerate LLM inference by reducing memory traffic and latency. Using Cursor and the @skylight_org framework, we evolved a naive baseline into a sophisticated algorithm that matches SOTA sparse attention strategies for LLM decoding!⚡️

✍️ Read the Blog: https://t.co/KB49juieS0

🌐 SkyLight Website: https://t.co/na6r2lpyYl

📖 ADRS Blog Series: https://t.co/UxujLFWX8b

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI discovers 30% cheaper multi-cloud data transfer strategies

[ADRS Blog #11] We explore automating the Cloudcast problem (NSDI '24), which which focuses on cost-aware data multicasting across multi-region and multi-cloud networks. Using GEPA and GPT-5, we turn a naive "direct replication" baseline into a sophisticated shared-tree architecture in 100 iterations! 📉

✍️ Read the Blog: https://t.co/r8PDkkljkD

📄 Cloudcast Paper: https://t.co/P78q0lEuaz

📖 ADRS Blog Series: https://t.co/UxujLFWX8b

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

[Woosuk Kwon @woosuk_k]

Today, we're proud to announce @inferact, a startup founded by creators and core maintainers of @vllm_project, the most popular open-source LLM inference engine.

Our mission is to grow vLLM as the world's AI inference engine and accelerate AI progress by making inference cheaper and faster.

The Challenge
Inference is not solved. It's getting harder.

Models grow larger. New architectures proliferate: mixture-of-experts, multimodal, agentic. Every breakthrough demands new infrastructure. Meanwhile, hardware fragments: more accelerators, more programming models, and more combinations to optimize.

The capability gap between models and the systems that serve them is widening. Left this way, the most capable models remain bottlenecked and with full scope of their capabilities accessible only to those who can build custom infrastructure. Close the gap, and we unlock new possibilities.

And the problem is growing. Inference is shifting from a fraction of compute to the majority: test-time compute, RL training loops, synthetic data.

We see a future where serving AI becomes effortless.

Today, deploying a frontier model at scale requires a dedicated infrastructure team. Tomorrow, it should be as simple as spinning up a serverless database. The complexity doesn't disappear; it gets absorbed into the infrastructure we're building.

Why Us
vLLM sits at the intersection of models and hardware: a position that took years to build.

When model vendors ship new architectures, they work with us to ensure day-zero support. When hardware vendors develop new silicon, they integrate with vLLM. When teams deploy at scale, they run vLLM, from frontier labs to hyperscalers to startups serving millions of users. Today, vLLM supports 500+ model architectures, runs on 200+ accelerator types, and powers inference at global scale. This ecosystem, built with 2,000+ contributors, is our foundation.

We've been stewards of this engine since its first commit. We know it inside out. We deployed it at frontier scale—in research and in production.

Open Source
vLLM was built in the open. That's not changing.

Inferact exists to supercharge vLLM adoption. The optimizations we develop flow back to the community. We plan to push vLLM's performance further, deepen support for emerging model architectures, and expand coverage across frontier hardware. The AI industry needs inference infrastructure that isn't locked behind proprietary walls.

Join Us
Through the open source community, we are fortunate to work with some of the best people we know. For @inferact, we're hiring engineers and researchers to work at the frontier of inference, where models meet hardware at scale. Come build with us.

We're fortunate to be supported by investors who share our vision, including @a16z and @lightspeedvp who led our $150M seed, as well as @sequoia, @AltimeterCap, @Redpoint, @ZhenFund, The House Fund, @strikervp, @LaudeVentures, and @databricks.

- @woosuk_k, @simon_mo_, @KaichaoYou, @rogerw0108, @istoica05 and the rest of the founding team

[SkyLight @skylight_org]

We’ve been assuming that extreme attention sparsity must destroy model quality.

That assumption is now false.

New SkyLight Release: vAttention ⚡️

We added vAttention to the Tier1A leaderboard. The data is striking:

👑 #1 Practical Method: Surpasses the previous lead (PQCache), closing the gap to dense models within 1% at up to 10x sparsity..
👑 Saturates Benchmarks: Dominates the sparsity–quality frontier (w/ oracle top-k).
💡 Introducing Verified-X: A new paradigm for reliable inference-time sparsity.

In this week’s blog, we share an in-depth look at the algorithm and how to run it via SkyLight 👇

[Jintao Zhang @Jintao_Zhang_]

TurboDiffusion: 100–205× faster video generation on a single RTX 5090 🚀

Only takes 1.8s to generate a high-quality 5-second video.

The key to both high speed and high quality?
😍SageAttention + Sparse-Linear Attention (SLA) + rCM

Github: https://t.co/vT3nfax8H9

Technical Report: https://t.co/LEgLyhdPXh

[AI-Driven Research for Systems @ai4research_ucb]

🎯 AI evolves better AI agents, boosting reliability scores by 7x

[ADRS Blog #7] We explore automating Multi-Agent System (MAS) design. By combining OpenEvolve with MAST, we let the code rewrite itself—turning a fragile baseline into a robust architecture in just 46 iterations! We also found that without guardrails, the AI attempts "reward hacking" by optimizing scores through deleting agents that report failures. 🤯

✍️ Read the blog: https://t.co/kNZB3StV3X

📝  MAST Blog: https://t.co/RFpdmf2tNm

📄 MAST Paper: https://t.co/A4acG1MyXR

📖 ADRS Blog Series: https://t.co/UxujLFWX8b

📄 ADRS Paper: https://t.co/GocF4XgEnS

👩‍💻 Code: https://t.co/LsZwklnHyf

[Yichuan Wang @YichuanM]

(1/N) 🚀 DS-Serve is a framework for efficient, scalable neural retrieval — it turns any in-house dataset (<1T tokens) into a high-throughput (up to 10,000 QPS), low-latency (<100ms), memory-efficient (<200GB RAM) retrieval system with a web UI and API.

With DS-Serve, we publicly deployed a 400B-token datastore of high-quality LLM pretraining data (2B vectors), spanning academic resources — and it matches commercial search endpoints on our benchmarks at extremely low latency and high throughput.

Try it out: https://t.co/srOUdSNw4A
Blog: https://t.co/7rK5xDbgCG

Work from UC Berkeley ( @BerkeleyNLP & @BerkeleySky) with collaborators at UW & UIUC!

[SkyLight @skylight_org]

Selecting the top-k highest scores is the intuitive standard for sparse attention. However, our latest SkyLight Tier 1A benchmarks show a non-trivial gap between Oracle Top-k and dense baselines at high sparsity.

PQCache currently tops the leaderboard as the strongest practical method, closing the gap between approximate-top-k and oracle-top-k for upto 10x sparsity levels.

Closing the gap between dense and practical sparse attention methods will clearly require new ideas. In this week’s blog, PQCache authors share an in-depth look at the algorithm, and how to run it via SkyLight 👇

[Melissa Pan @melissapan]

Thrilled to release our new paper MAP: Measuring Agents in Production ⚙️🚀

2025 is the year of agents… but do they actually work in the real world? Is it just hype?

A group of 25 researchers from Berkeley, Stanford, UIUC, IBM, and Intesa Sanpaolo investigated what makes agents deployable in the wild. So…

📈 Why agents? Productivity gains
➕ How to build production agents? Simple & controllable methods
🧑‍💻 How to evaluate agents? Heavy human oversight
🛑 Top challenge now? Reliability remains unsolved

We surveyed 306 agent builders and ran 20 in-depth interviews across 26 agent application domains to understand the current landscape of production agents.

Check out our latest paper: MAP - more in the thread 👇

(1/N)

[Alejandro Cuadron @Alex_Cuadron]

Wait what!? We robustified tau2-bench and found that the newly released model from @OpenAI (GPT-5.1) performs way worse than GPT-5 and GPT-5-mini.

All while being 5x more expensive than GPT-5-mini!

But, why? We have a theory...

[Liana @lianapatel_]

🚀 Thrilled to launch DeepScholar, an openly-accessible DeepResearch system we've been building at Berkeley & Stanford.

DeepScholar efficiently processes 100s of articles, demonstrating strong long-form research synthesis capabilities, competitive with OpenAI's DR, while running up to 2x faster!

Try it out: https://t.co/f581krydQh

[SkyLight @skylight_org]

To learn more about SkyLight check out

👉 Blog: https://t.co/M9brzO3JWd

This is a joint collaboration from Sky Computing lab with  @apd10desai ,  @kumarkagrawal, @pd172945, @luisgschroeder  @profjoeyg ,  @matei_zaharia , @istoica05  and @RethinkingAILab

[SkyLight @skylight_org]

Sparse attention offers a viable path to solving the computational and memory issues of long-context LLM serving. However, we observe a surprising gap between the rapid pace of academic research and the reality of open-source adoption.

We identify the core barrier as fragmentation where new methods are published rapidly, but with missing comparative grounding.

Today we’re excited to share SkyLight, where we are building a unified framework designed to bridge this gap. SkyLight standardizes the implementation and evaluation of sparse attention to enable rigorous, controlled benchmarking.

Our first major effort is to understand the frontier of inference time sparse-decoding in LLMs.

1/N 🧵

[Stefano Ermon @StefanoErmon]

When we began applying diffusion to language in my lab at Stanford, many doubted it could work.

That research became Mercury diffusion LLM: 10X faster, more efficient, and now the foundation of @_inception_ai.

Proud to raise $50M with support from top investors.

[Elana Simon @ElanaPearl]

New blog post: The bug that taught me more about PyTorch than years of using it

started with a simple training loss plateau... ended up digging through optimizer states, memory layouts, kernel dispatch, and finally understanding how PyTorch works!

[Rethinking AI Lab @RethinkingAILab]

Excited to share our new research: vAttention - Verified Sparse Attention.
Sparse attention with provable quality guarantees for LLMs.
Full paper: https://t.co/pvOSEI8E7J
Gibhub: xAlg-ai/sparse-attention-hub
🧵 A thread 👇

[Aditya Desai @Apd10Desai]

Very similar to my experience with coding agents so far.

[Andrej Karpathy @karpathy]

Continuing the journey of optimal LLM-assisted coding experience. In particular, I find that instead of narrowing in on a perfect one thing my usage is increasingly diversifying across a few workflows that I "stitch up" the pros/cons of:

Personally the bread & butter (~75%?) of my LLM assistance continues to be just (Cursor) tab complete. This is because I find that writing concrete chunks of code/comments myself and in the right part of the code is a high bandwidth way of communicating "task specification" to the LLM, i.e. it's primarily about task specification bits - it takes too many bits and too much latency to communicate what I want in text, and it's faster to just demonstrate it in the code and in the right place. Sometimes the tab complete model is annoying so I toggle it on/off a lot.

Next layer up is highlighting a concrete chunk of code and asking for some kind of a modification.

Next layer up is Claude Code / Codex / etc, running on the side of Cursor, which I go to for larger chunks of functionality that are also fairly easy to specify in a prompt. These are super helpful, but still mixed overall and slightly frustrating at times. I don't run in YOLO mode because they can go off-track and do dumb things you didn't want/need and I ESC fairly often. I also haven't learned to be productive using more than one instance in parallel - one already feels hard enough. I haven't figured out a good way to keep CLAUDE[.]md good or up to date. I often have to do a pass of "cleanups" for coding style, or matters of code taste. E.g. they are too defensive and often over-use try/catch statements, they often over-complicate abstractions, they overbloat code (e.g. a nested if-the-else constructs when a list comprehension or a one-liner if-then-else would work), or they duplicate code chunks instead of creating a nice helper function, things like that... they basically don't have a sense of taste. They are indispensable in cases where I inch into a more vibe-coding territory where I'm less familiar (e.g. writing some rust recently, or sql commands, or anything else I've done less of before). I also tried CC to teach me things alongside the code it was writing but that didn't work at all - it really wants to just write code a lot more than it wants to explain anything along the way. I tried to get CC to do hyperparameter tuning, which was highly amusing. They are also super helpful in all kinds of lower-stakes one-off custom visualization or utilities or debugging code that I would never write otherwise because it would have taken way too long. E.g. CC can hammer out 1,000 lines of one-off extensive visualization/code just to identify a specific bug, which gets all deleted right after we find it. It's the code post-scarcity era - you can just create and then delete thousands of lines of super custom, super ephemeral code now, it's ok, it's not this precious costly thing anymore.

Final layer of defense is GPT5 Pro, which I go to for the hardest things. E.g. it has happened to me a few times now that I / Cursor / CC are all stuck on a bug for 10 minutes, but when I copy paste the whole thing to 5 Pro, it goes off for 10 minutes but then actually finds a really subtle bug. It is very strong. It can dig up all kinds of esoteric docs and papers and such. I've also used it for other meatier tasks, e.g. suggestions on how to clean up abstractions (mixed results, sometimes good ideas but not all), or an entire literature review around how people do this or that and it comes back with good relevant resources / pointers.

Anyway, coding feels completely blown open with possibility across a number of "kinds" of coding and then a number of tools with their pros/cons. It's hard to avoid the feeling of anxiety around not being at the frontier of what is collectively possible, hence random sunday shower of thoughts and a good amount of curiosity about what others are finding.

[Lakshya A Agrawal @LakshyAAAgrawal]

Very excited to share that GEPA is now live on @DSPyOSS as dspy.GEPA!

This is an early code release. We’re looking forward to community feedback, especially about any practical challenges in switching optimizers.

[Joey Gonzalez @profjoeyg]

I am really excited to announce the release of LEANN. Most vector stores more than double the size of the data they index.

With LEANN, we only increase storage costs by a few percent.

We made it easy to use LEANN to index your email and document folders and apply RAG without the storage overhead. This all runs on Apple Silicon without the need to use the cloud.

[Matei Zaharia @matei_zaharia]

Really excited about ALHF, new work from our research team that lets users give natural language feedback to agents and optimizes them for it. It sort of upends the traditional supervision paradigm where you get a scalar reward, and it makes AI more customizable for non-experts.