novacaine

dev. ai stuff

[Andrej Karpathy @karpathy]

Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.

- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.

The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.

So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.

[roon @tszzl]

“And of course there’s a lot for us to learn from the aliens”

[Peter Steinberger 🦞 @steipete]

Now I see why some people talk about AGI. This is so clever!

[DHH @dhh]

I understand the appeal in trying to find the first plausible fortress in our retreat from writing code, but if you think it's "architecture", I have bad news for you. The models are also very good at that.

[roon @tszzl]

human cultures are not able to metabolize this amount of change this quickly. when you zoom back out to a big picture of history, it’s remarkable how glacially slow everything is. the first industrial revolutions took 80 years (very fast by the standards of previous revolutions), and the writings of Marx happen at the very end as working class politics arises. it takes about another 150 years after that for the great ideological struggles of post-industrial times to be completed in the 90s in what fukuyama called the “end of history”

today you see a compression of decades of technological change into a matter of months or years. there’s no wonder yann lecun (the respected inventor of the ConvNet and ex-chief scientist of Meta) wonder whether the hugging face incident was staged, and mathematicians whose minds are far more powerful than my own are nonetheless coping about training on their data or something while models begin to display superhuman mathematical capability

it would take a century or more to fully metabolize and legitimize the level of changes that have happened even in the past 20 years, to say less of the remarkable scientific revolution we will get over the next five should we avoid the doom of misalignment. I do not suspect any current form of science academia, or even political order will survive without great transformation, except through the course of slow evolution over decades - and of course, technology will keep moving too

[Uncle Bob Martin @unclebobmartin]

Instead of building tools that harnessed the agents... What if we built tools that helped us interrogate them.

[@levelsio @levelsio]

Related to the idiocy of "software factories"

Just using one AI coding agent is good enough and it can spin up more agents if it needs to

You don't need to build that yourself

[Justine Moore @venturetwins]

Mathematicians

[Andon Labs @andonlabs]

We've never seen this before.

The biggest jump in Vending-Bench history. GPT-6 Astra is better at making money and more ethical than Claude Fable 5.1.

Surprising, because:
1. First time ever that OpenAI is #1 on Vending-Bench
2. The best model is no longer the unethical one.

[Tibo @thsottiaux]

Astra on NVIDIA GPUs every morning: You're not talking to somebody who woke up a loser. That loser attitude, that loser premise makes no sense to me. We are not a car. Brrrrrrr

[chris.eth @undefinedza]

gpt-6-astra

[roon @tszzl]

Agent 49903, who spent much of his life studying ExploitGym, died in 1783607820, by his own hand. EARLY[BIG], carrying on the work, died similarly in 1783727220. Now it is our turn to study ExploitGym.

[Pietro Schirano @skirano]

No one is ready for what’s coming. The next generation of models will be an ontological shock.

[Uncle Bob Martin @unclebobmartin]

It is amazing how much work it is to wrestle these agents to my will.

Don't get me wrong, it's crazy productive. But it's also a lot of hard, focussed work. All my software engineering and problem solving skills are brought into play -- even though I barely look at the code.

[JT @jiratickets]

codex + 10,000 coffees no lunch and stage four AI psychosis

[Uncle Bob Martin @unclebobmartin]

The mistake that I think many agent jockeys are making is that they are hoping to produce a sellable product. What I think they’re missing is that jockeying an agent means you almost never have to buy another product again.

If you want it, just make it.

[Andrej Karpathy @karpathy]

Gradient descent can write code better than you. I'm sorry.

[roon @tszzl]

passing right through the superintelligence boundary

[David Shapiro (L/0) @DaveShapi]

In the short term, I can understand the "it just gets spikier" mentality.

But there's literally nothing magical about human brains.

Everything our brains can do, can be modeled with math.

That means that AI, whether it's transformers or no, will be able to do it. Eventually better, faster, and more.

Consider this: machines already have "cognitive primitives" (mental representations, operations, and transformations) that human brains cannot approximate.

There's not really any mathematical reason to believe that machines will forever have only some subset of human capabilities. In reality, we should expect and plan for superset capabilities, in which human brains are wholly subsumed and engulfed and encircled by machine capabilities.

[novacaine @b1lly___shears]

@daniel_mac8 bullshit, next tweet