Adya Srivastava

curious about too many things, reducing epistemic uncertainty one questionable hypothesis at a time. currently in my pretraining era, please hold.

[Adya Srivastava @adyasrivastava2]

Very True. Running 200 agents on internal models is a very different game from what most of us get to use. We really need better ways to assess how intelligence generalises and where reasoning actually breaks, beyond benchmark scores.

[Thomas Jiralerspong @tomjiralerspong]

Daily reminder that model capabilities are jagged...

[Adya Srivastava @adyasrivastava2]

I’m exploring this through research on reasoning evaluation and mechanistic interpretability: identifying failure mechanisms, testing interventions, and using those insights to build AI systems that adapt their reasoning to each task. Would love to connect.(2/2)

[Adya Srivastava @adyasrivastava2]

Buying custom RL envs or bigger weights won't solve enterprise AI. The missing moat is an intelligence layer that diagnoses why models fail, not just that they failed. Causal analysis + adaptive inference lets us scale compute and change harnesses per query. (1/2)

[Daniel Rupawalla @danielrupawalla]

i don't think the average person / company will be able to make useful RL envs.

the bet companies that want to own their own intelligence need to make is either that open-source models will get good enough to just do things out of the box or trust in data/training companies to train increasingly smart models for their specific use case. i could see a world where distillation as a service becomes popular / maybe some internal engineers can figure this out, but making frontier-level tasks for a model like kimi-k3 is non-trivial, and it will only get harder to do this as models progress. you should be out-sourcing your task creation to a company that knows how to do it well. if you're training a really large model and/or need it for a very high traffic use-case, it's not really worth the risk to half-bake a solution, buy really good weights once.

[Adya Srivastava @adyasrivastava2]

@timhuyn Both evaluations reveal different things about LLMs and their training. There’s an equality vs equity question, but also generalizability. If a model needs an expert built harness to perform well, do those findings generalize to the wider user base? The intended audience matters.

[Adya Srivastava @adyasrivastava2]

Really interesting work on ICA! I’m curious whether these features capture the computations models actually rely on, or just patterns in their activations. Would intervening on them reveal the difference?

[Furong Huang @furongh]

SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?

In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.

Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!

But there's another motivation behind this work that I care about even more.

I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?

With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.

Interpretability shouldn't end with understanding. It should empower us to act.

And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.

🌹 Try it yourself:
pip install icalens
https://t.co/5jaTp29eX9

Curious to hear what the mechanistic interpretability community thinks!

[Adya Srivastava @adyasrivastava2]

@tomjiralerspong Very true!! It’s gonna be the most exciting time in interpretability now.

[Adya Srivastava @adyasrivastava2]

Too impt to dump in saved and not repost!!

[Alex Wang @heyyalexwang]

the post derives all of this step by step, with interactive figures where you can watch memories pile up yourself

plus pointers to the latest work on memory like recent works!

https://t.co/Bbrj7QpLZJ

[Adya Srivastava @adyasrivastava2]

I think we underestimate how much compute and setup matter when evaluating AI.
OpenAI reportedly spent $15M to solve a $1M problem, with 1000 agents working together. Most of us use a $20/month subscription.
We need to rethink how we compare intelligence.

[gabriel @gabriel1]

ai models solved some of the hardest unsolved math problems and most people still think it struggles on multiplication

interesting times we live in, the biggest alpha to everything has never been so easily accessible but hard to understand. you just need to tell ai what you want

[Adya Srivastava @adyasrivastava2]

@samongaro_ CLARITY. Not everyone knows what’s worth building, why, and for whom.

[Adya Srivastava @adyasrivastava2]

Nick and Neel’s exchange got me thinking: can an attribution graph predict where a model’s reasoning fails?
Test the same problem across 20 phrasings. Can it predict errors on unseen ones?
Then intervene on the suspected circuit. Does this improve accuracy on unseen phrasings?

[Neel Nanda @NeelNanda5]

This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.

I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!

Similarly, my impression is that SAEs generally don't find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.

And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

[Adya Srivastava @adyasrivastava2]

@OfficialLoganK I think a lot of our misery comes from struggling to accept what is. And true acceptance is harder than most people realise.

[Adya Srivastava @adyasrivastava2]

Came across this May 2025 post from @karpathy while researching LLM reasoning.
If models learn strategies from mistakes, how do we know those strategies generalize beyond familiar problems?
I want to test this by changing prob surfaces and probing the mechanisms behind the ans.

[Andrej Karpathy @karpathy]

We're missing (at least one) major paradigm for LLM learning. Not sure what to call it, possibly it has a name - system prompt learning?

Pretraining is for knowledge.
Finetuning (SL/RL) is for habitual behavior.

Both of these involve a change in parameters but a lot of human learning feels more like a change in system prompt. You encounter a problem, figure something out, then "remember" something in fairly explicit terms for the next time. E.g. "It seems when I encounter this and that kind of a problem, I should try this and that kind of an approach/solution". It feels more like taking notes for yourself, i.e. something like the "Memory" feature but not to store per-user random facts, but general/global problem solving knowledge and strategies. LLMs are quite literally like the guy in Memento, except we haven't given them their scratchpad yet. Note that this paradigm is also significantly more powerful and data efficient because a knowledge-guided "review" stage is a significantly higher dimensional feedback channel than a reward scaler.

I was prompted to jot down this shower of thoughts after reading through Claude's system prompt, which currently seems to be around 17,000 words, specifying not just basic behavior style/preferences (e.g. refuse various requests related to song lyrics) but also a large amount of general problem solving strategies, e.g.:

"If Claude is asked to count words, letters, and characters, it thinks step by step before answering the person. It explicitly counts the words, letters, or characters by assigning a number to each. It only answers the person once it has performed this explicit counting step."

This is to help Claude solve 'r' in strawberry etc. Imo this is not the kind of problem solving knowledge that should be baked into weights via Reinforcement Learning, or least not immediately/exclusively. And it certainly shouldn't come from human engineers writing system prompts by hand. It should come from System Prompt learning, which resembles RL in the setup, with the exception of the learning algorithm (edits vs gradient descent). A large section of the LLM system prompt could be written via system prompt learning, it would look a bit like the LLM writing a book for itself on how to solve problems. If this works it would be a new/powerful learning paradigm. With a lot of details left to figure out (how do the edits work? can/should you learn the edit system? how do you gradually move knowledge from the explicit system text to habitual weights, as humans seem to do? etc.).

[Adya Srivastava @adyasrivastava2]

OmniMesh is still an early project, and I’m looking to learn from people who’ve worked on similar problems.
I’d love feedback on the approach, what seems impractical, and what I should explore next.
Live project: https://t.co/6mRmOXyZSu

[Adya Srivastava @adyasrivastava2]

1B+ Android earthquake alerts is incredible. But what happens next?
Victims can’t always call for help. Networks fail. Rescuers lose visibility.
I built OmniMesh to help phones detect distress and relay SOS offline.
Demo: https://t.co/1IkJkMVszz
Code: https://t.co/SDyhpZp4FF

[Sundar Pichai @sundarpichai]

Since launching in 2021, our Android Earthquake Alerts System has sent 1B+ alerts. Now we’re expanding the system from 98 to nearly 200 countries and regions.

Proud how this technology can help in moments that really matter. As one example, when the devastating earthquakes hit Venezuela this year, millions of people outside the epicenter got alerts and could take cover before the shaking began.

Read more below https://t.co/UEb7DC69Pi

[Adya Srivastava @adyasrivastava2]

@elonmusk Incredible…..Transformers next??

[Adya Srivastava @adyasrivastava2]

Just cos we have calculators doesn’t mean we stop learning addition.
DSA teaches you how code and systems work and builds your logic. You need that foundation to know what to build, how to build it, and how to guide AI agents.

Still worth learning, imo.

[Nandini @N_and_ni]

Is it still worth learning DSA ?
Is it still worth learning DSA ?
Is it still worth learning DSA ?
Is it still worth learning DSA ?
Is it still worth learning DSA ?

[Adya Srivastava @adyasrivastava2]

@MicrosoftLearn Any good courses on coding effectively with AI agents? Something that covers orchestrating multiple agents, understanding and reviewing their code, and learning as you build, rather than blindly accepting whatever they generate.

[Adya Srivastava @adyasrivastava2]

@paulg Interesting take. Have AI agents proven themselves on real-world sales tasks? Once API and oversight costs are factored in, could a sales expert using AI be cheaper than a founder managing agents instead of building?

[Adya Srivastava @adyasrivastava2]

Before we trust AI agents to replace entire roles, we need benchmarks that measure outcomes in the real world, not just task completion in controlled settings.

Can an agent do the work, judge its own output, and recover when things go wrong? And at what cost of human oversight?

[Paul Graham @paulg]

Did office hours yesterday with a very smart but nerdy founder. When I suggested he might want a more outgoing cofounder to do sales, he said he didn't, because most of his sales would be to agents. I had to admit he could be right.

[Adya Srivastava @adyasrivastava2]

@Sarthakhq @adxtyahq @browsercompany I like Arc coz switching between profiles and workspaces is seamless. Keyboard shortcuts, folders, favourites and pinned sites make managing tons of tabs easier. It uses more RAM than Chrome, and with development mostly on life support, I doubt we’ll see major new features.