Hiranmay Darshane

deep learning and large language models. football (banter) fan. 19.
prev: research intern at

[Hiranmay Darshane @hdarshane]

In Sept 2024, o1 surprised many purists who thought inference-time scaling for LLMs was through MCTS.
What if a connection exists, just implicit? What does it imply?
New post: "Squint enough and RLing CoT reasoners is approximable as Monte Carlo Tree Search policy learning." 🧵

[Hiranmay Darshane @hdarshane]

@exhaze @hrishioa lol ill take it

[Hiranmay Darshane @hdarshane]

What is left for us to do?

It might be the case that broadly, only two or three technically meaningful problems are left for us to solve (ironically many more OOMs of non-technical ones!). I'll limit myself to "capabilities" stuff, safety deserves its own thoughtstream.

We're already at the foothills of superintelligence. As models keep improving, most open problems get solved as a side effect, and work spent on them leaves little trace behind after the solution arrives.
A small number of problems either resist that pressure or only become solvable because of it, and the people who solve them, or who build the frame everyone else works inside, get to bask at least in some satisfaction/vindication of being correct or just doing what was important.

Everyone else contributes something too, if only by mapping dead ends, but many of them may end up regretting their effort could have gone somewhere more durable with better foresight. At the end of the day, it is desirable to end up knowing one was correct and one’s work wasn’t “sniped” by just better methods in general that solved one’s work as a downstream thing/side-effect.

That makes picking the problems/directions the main decision, and getting it wrong is at the very least emotionally very expensive.

1) continual learning (or at a more “tamable” level of solving: operating over unbounded context in general). IMO this gets solved by training models to manage their own context at direct KV level (https://x[.]com/RulinShao/status/2105282444270448647?s=20) + cartridges-like offloading.

We might never solve/require true parametric continual learning because cartridges/context management and the suchlike may suffice for true “learning on the job” at a per-deployment level, and at a “central” level, newer pretrains (plus some post-release post-training) probably denecessitate true parametric continual learning.

2) sample efficiency/generalisation. This is somewhat tied in with the first one i.e. if you want to learn “new” things encountered upon deployment (which are inherently rare), you should be able to learn and extract their value efficiently.

One perspective I’ve learned to value is to think of/attribute generalisation unlocks in terms of “levers of generalisation”.
Ex: a) Chain-of-thought which is almost like metacognition that transfers to almost everything.
b) envs/RL on tools in general which encourages models to poke around more and thus teach how to use a fixed set of tools to interact with/make sense of the environment.
c) more data in general, since you are basing thoughts/actions off a wider set that you can interpolate/transfer from.

Relating to 1) you can imagine something like memory consolidation is pretty important and serves as a lever in that it relates recent/online experiences to your wider experiences/thoughts and how they relate to each other.

Continuing that thought, our RL methods right now are still low-compute and rudimentary, correctness reward on a rollout transferring/generalisation widely is contingent on super shared/general circuitry, etc. (which clearly exists and we can see generalisation to an appreciable degree), but we haven’t hit the ceiling yet on methods that allow the model to maximise the utility of learning new things. stuff like:
“Oh I just learnt this, can I go back and try out the same method on xyz things I failed on?”, “Oh wow this was the trick! Where else can I apply it to?”.

This might be too hand-wavy but the general idea is to start with the (reasonable) assumption that whatever generalisation we see is a happy accident (of the way NNs actually grow themselves + the data generation process of the world) and there’s probably space for us to intervene to maximise that.

One general framing I like is that we haven’t explored the “low samples, high compute” quadrant of learning methods yet (and that isn’t restricted to just RL but also better arch + optim, etc.) (x[.]com/moultano/status/2106912955459920309)

3) better/faster/cheaper systems: there was a nice post called “The many ways that digital minds can know” (https://t.co/YLYajgH2fX).
One of its merits was motivating the bitterness of how NNs can learn. I think there’s another post due titled “The many ways that digital minds can do”.

In general, many/most tasks of the world are amenable to a small horde of reasonably-intelligent agents having a crack at them.

Some tasks are “make the tower taller” i.e. inherently unbounded with high returns to marginal dakka (effort, or something like # of parallel attempts, as opposed to just intelligence).
Other tasks are like “make the ball rounder” with diminishing (or zero) returns to more dakka/intelligence. It seems to me that we underestimate the % of tasks that are the former.

We continue to underestimate the importance of boring brute-force scaling of this style relative to something like “capability unlocks”. If you believe in this, you would straightforwardly believe that cheaper/more efficient compute (in a general interpretation of intelligence/$ or per watt) is one of the most important problems to solve for, and stuff like better test-time parallelisation/coordination.

[Hiranmay Darshane @hdarshane]

@joodalooped does not explain all of it, there are incentives to throw lots of compute at RL but disk will keep being filled only asymptotically

[Hiranmay Darshane @hdarshane]

Many quasi-"purists", me included, have seen the general meta of the industry w.r.t. RL envs with some snobbishness and flippance: "eww no the path to AGI should not be riddled with a horde of data contractors hand-building envs that pass certain specifications"

But, I have had some recent thoughts/muses that helped reconcile my own position (the thoughts excluding the subjective bitterness, but the actual inability to understand why this is the meta we converged on): It is well-understood that model capabilities (and gains therein) are fractal-like and follow long-tails (see img attached).

Imagine capabilities work as seeing the circle-packed disk below (Apollonian gasket) and trying to fill in big parts. The problem is that as parts get smaller and smaller, more specific and more scattered, it is an understandable concern to bet against super general uplifts (since the major part of the disk is filled in already, and the transfer/generalisation that we observe is not amazing).

More importantly, the bottleneck in such an “almost-filled-in-completely” regime is figuring out the uncolored parts in general. RL companies work similarly with the intent being to find cases/tasks where models have a low passing rate. The great evals are built in similar spirit and thus the “every good eval is an RL env” observation many make.
Of course, a bitter purist (like me) looks at this and still says "it's not general enough", but at least it makes perfect sense as to why it exists. In general, figuring out (and hillclimbing) the (ever decreasing) share of tasks that models can't do is a necessary and perfectly honorable pursuit.

But one keeps hoping that we see something more beautiful. Automated env design (adversarial setups will return), algorithms that generalise and transfer OOMs better (taking the "low samples, high compute" quadrant seriously), a new "meta", etc. etc.

[Hiranmay Darshane @hdarshane]

at the end of the day, our only job is to build great circuits at train-time, and to ensure those are found and exploited at test-time.

[Hiranmay Darshane @hdarshane]

going to be there! please join us!
we're also looking for more speakers/presenters, DM if interested.

[calvin @calvingenuity]

Mumbai’s first alphaXiv Paper Club is happening on Nov 7!

We’ll discuss papers on Continual Learning. We hope to meet monthly and build a research community in the city.

Doing AI research or getting into it? Join us:
https://t.co/QANTDj4iPe

[davinci @leothecurious]

if this meme was a paper
https://t.co/lptTcHVkwa

[Jan Kulveit @jankulveit]

Few years ago, there was just a handful of economists seriously thinking about AI (@akorinek,@pawtrammell,@erikbryn,Carl Shulman & few of their collaborators) and I was like 'this is insane, there should be 10x as many'.

Now it's ~10x-30x more and frankly it is disappointing so far. Majority of the new economists bring a mindset and frames which are something between unhelpful to directly harmful for understanding AGI
- not taking AGI seriously / "AI as a normal technology"
- merchants of cope dynamic (in high demand by the labs)
- labour/capital mindset
- people were wrong about risks of tech transitions in the past and by induction they are again
etc...
coupled with epistemic arrogance. Sorry for being very blunt, by my impression is part of the problem is economists are often used to be in the broader context of humanities where they are the most quantitative and in some ways the smartest lot in various rooms, and others disagree with them for bad reasons. But this is _not_ the case relative to AI safety a few years ago!

[Hiranmay Darshane @hdarshane]

@RobertTLange Opus 5.5 may do even better

[Hiranmay Darshane @hdarshane]

@aengus_lynch1 @AnthropicAI @theoremlabs amazing work!

[Hiranmay Darshane @hdarshane]

A vacation is an act of indefinite optimism

[Devvrit @Devvrit_Khatri]

On policy methods change non-principal direction, and hence also avoid forgetting. We take this insight and restrict any finetuning algo to do the same. Result: almost no forgetting, while still learning the target task well.
Also scales well with # params & data.

[Hiranmay Darshane @hdarshane]

@cs_serdar unbelievable work and speed ❤️

[Hiranmay Darshane @hdarshane]

This post got me thinking.

Must the birth of god be plagued by so many maladies?

It’s becoming clearer and clearer that public sentiment on AI is souring and many large groups of the population are questioning whether their obsolescence is looming, and are worried about all that may come later.

We can talk about how it’s the greatest thing ever and potentially the last meaningful majorly human invented object (I STRONGLY agree). To some it has been super intuitive that such a transition is going to be anything but stable, while others, completely fairly, may have expected steadier transitions.

Impact is steadily revealing itself in sentiment surveys, and some employment statistics (https://t.co/0kwUPOH0oI, https://t.co/5eL2rxVAfc). In general, it seems intuitive that entry-level hiring may reduce drastically since these tasks are now automatable and the overall automatability of the white-collar economy (measured by say length-horizon and total tasks exposed) will clearly keep increasing.

The doubt is informed. Why, then, does informed doubt not turn into anything? There was an old experiment (Latané, B., & Darley, J. M. (1968)) that did the following: put someone alone in a room, start to pump smoke under the door, and then about three in four get up and report it. Put them with two confederates instructed just to sit calmly and it drops to around one in ten. The subjects see the smoke just as clearly. There’s a great Yudkowsky post (https://t.co/ozxmh7LXih). A fire alarm in a sense does not tell anyone something they couldn’t already see. But it helps everyone know that everyone knows. A normaliser of beliefs, in a sense. It’s even more important to have mechanisms to normalise beliefs when the beliefs get crazier and harder to believe (relative to our “nothing ever happens” priors and expectations).

An issue is that the confederates are incentivised to stay silent not just by “trying not to seem crazy”, or self-interest derived from being within the industry, but just the stakes involved if we try to reverse trends. AI investment is now estimated at roughly a third of US GDP growth this year, and all the doomposts of what happens if the “bubble bursts” have pretty damning consequences to make. My point here is to say, sometimes the “normaliser of beliefs” (here being okay with whatever might happen) is often just macro inevitability like this. Economic stakes are high, no lab can afford to slow down unilaterally (and each is probably cynical of adherence to mutually negotiated “slowdowns”). A sincere belief and being-okayness in inevitability is probably rational.

But is it righteous?

I’m personally pretty worried about a “software-only singularity” or broadly a “jagged singularity”. Think AI takes over the majority of (not restricted to) entry-level jobs in a couple of sectors, with others remaining mostly untouched. The kind of dynamics that would play out inter-occupationally seem pretty bad. In the limit, when intelligence is not jagged and is uplifting most tasks uniformly, I’m also pretty worried about what happens to people (mostly young kids who are potentially fresh grads at the time) who have worked all their life working through a system, with expectations and perceptions of the good rewards that lie at the end of it, mostly shaped by others with models from super outdated times and circumstances.

That’s just one scenario on top of my mind (not weighted by likeliness, etc.). There’s probably a longer list. If this really is the last major human invention, then after it most value would come from owning the thing rather than doing the work, and it’s pretty unclear who ends up owning it (or how the rest of us get a share, other than by someone’s generosity). Essentially the social ladder freezes. Work has been the ladder into the middle class for a whole generation of families, and may stop being a viable uplifter.

Another perspective is that we need to start upweighting and caring more about the long-tail risks (utter societal mutiny, mass unemployment just within a few years, the complete end of all social mobility). As unlikely as they are, the expected value is just unimaginably negative. Even then, they don’t look super unlikely either. A major change over the last couple of months is the realisation that paths to say: takeover risks, mass unemployment are super real and will keep becoming so in the future.

Even on expected-value terms this looks bad, since the risks don’t look super unlikely anymore. But I think maybe a better frame is maximin over maximax: judge a path by how its worst-off fare rather than by how glorious the best case is. That’s especially true when, as here, we can’t really estimate the probabilities.

And then some personally uncomfortable ones. Many of us, even if in extremely small ways, have contributed to capabilities research ourselves, so part of “is it righteous?” is a question about whether we’re the dumb calm strangers in the smoke room. Will people a few decades down the road think researchers were complicit for not being more vocal about the risks and/or going forward like usual anyway?

Yudkowsky’s original post had a good point that there will be no “alarm” for AGI-related risks and transitions. In my perspective, the risk vectors are just too many and super non-linear, so the “no alarm” take is infinitely more valuable today.

What would building one look like? In general how do we positively shape progress and “birth the god” without maladies plaguing? We probably need reliable alarms that help people know what’s up, although this might seem like central planning that is likely to fail.

(1) One important perspective is that we need to delinearise the current axes of terminologies (name-calling, really) w.r.t. “doomers” and “optimists”. It is probably okay to be hopeful about the technology (and even contribute to capabilities) while being super vocal about the risks that are coupled. People from labs speaking their hearts out probably helps a lot. I was pretty moved by this line of reasoning from Dan Selsam’s statement (paraphrasing and extending): One can want the glorious renaissance, having worked for it one’s whole career, and it breaks one’s heart to see the possibility that one has to forgo it. But what’s the point if we get there with super risky trades and generally just via ways that make people suffer (or takeover risks that Selsam mentions)?

(2) Require firms to attribute layoffs. Many recent layoffs have been swept under “overhired during post-COVID ZIRP and now cutting down” while they may be AI-related. This is a huge disservice and in a sense is spoofing a valuable alarm.

(3) How should this impact and inform actual capabilities research?

(3a) The jagged frontier is partly something researchers contribute to. Entry-level software went first largely because code comes with quick verifiers (tests pass or they don’t), which makes reinforcement learning work well there. If the development on the frontier is done by targeting some evals and then assessing how well they are doing w.r.t. them, every benchmark and reward signal you build pulls some occupation’s tasks across the line sooner.

(3b) Some would say we should focus on “augmentation”. It seems pretty crazy to think of AI systems trained to “augment” rather than replace. How do you even do that? Get systems to act like they don’t know stuff and thus seek human intervention/escalation? This seems like an open problem. In general, our terminology and meta-thought within the field is addressing “automation” and “replacement”, not augmentation. While very ironically, there seems to be (p=0.4) a case where the latter may result in more immediate productivity gains (complete automation may get bottlenecked on “straddler” hard-to-learn tasks, while augmentation is a quick uplift all across). https://t.co/wTcjyutaMz. Maybe augmentation vs automation aren’t super cleanly separable axes, but this remains worth thinking about.

(4) Do we need to stop/pause? This warrants a serious discussion. There have been some good pieces about how many benefits can be realised even during a pause (https://t.co/NMMv3k1u1Z) and in general we seem to be in a capabilities overhang scenario (i.e. we can spend many years just working through harvesting the current fruit only) (https://t.co/8jGHxCVVln).
Other stuff like protecting the junior pipeline, insuring against transitions, moving the tax base (as labour’s share falls) and perhaps better redistribution schemes. This is probably for another post.

The most famous maladies of birth, the childbed fevers in old maternity wards, turned out to be carried by the doctors’ own hands, and the fix was washing them. Maybe the birth of god doesn’t have to be plagued by so many maladies. Maybe it is so inherently if unchecked, but let’s not pretend we have no agency. It may seem intuitively unlikely that simple interventions meaningfully guide such “godly” transitions, but childbed fever also looked like fate until someone noticed whose hands it was on. Most of the maladies might just need boring, unglamorous hygiene, and a bunch of people willing to stand up and insist on it.

[Joe Weisenthal @TheStalwart]

AI IS AMAZING

I know it's hip to worry about the decline in book reading and how chatbots will impoverish our brains.

But TBH, this is not at all what I see. I wrote about how we're in this wild moment for intellectual flourishing and how all of us are the children of dancers.

[Hiranmay Darshane @hdarshane]

@ryangr Banger

[Hiranmay Darshane @hdarshane]

@SimulatedAnneal Hi Xingjian, I agree! but wasn't involved on zero-order/ES myself... tagging the guys that work on those ideas at Q!
@industriaalist @bishmdl76 @cs_serdar @akshayvegesna

[Hiranmay Darshane @hdarshane]

bitter and tasty as hell
wow insane work

[Xingjian Bai @SimulatedAnneal]

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?

We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.

In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.

📝 https://t.co/augwW3iFsB
💻 Try it on your task: https://t.co/NdQNekaHoc
🧵

[Hiranmay Darshane @hdarshane]

typa stuff u talk about as a breakthrough 10..20...30 years later

[Hiranmay Darshane @hdarshane]

I like both works, IMO bitter/blackpilled perspective is that standard general scaling can only asymptote towards a noise ceiling, after which ugly per-case work like building RL envs necessitates itself

[Hiranmay Darshane @hdarshane]

@willcb https://t.co/DaClWQxEmX
ayo

[Xingjian Bai @SimulatedAnneal]

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?

We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.

In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.

📝 https://t.co/augwW3iFsB
💻 Try it on your task: https://t.co/NdQNekaHoc
🧵