Martin Convery

[Martin Convery @MartinConvery9]

the king is dead.

[Daniel Lockyer @DanielLockyer]

New record for Stack Overflow:

<1000 questions posted last month

[Martin Convery @MartinConvery9]

@StanphylCap dario?

[Martin Convery @MartinConvery9]

@trbdrk I still like the low poly look:

[Aidan Clark @_aidan_clark_]

Whatever you believe about the world I think we can all agree that P(P = NP) is increasing....

[David Hines @hradzka]

we regretfully announce the AI slop’s tune is in fact a banger

[François Chollet @fchollet]

String theory is what happens when an intelligent system is starved of sensory input and starts hallucinating. Intelligence needs grounding.

For this reason, intelligence is practically never the bottleneck to solving impactful problems. The bottleneck is iterated contact with reality.

[François Chollet @fchollet]

The real world abounds with recursively self-improving systems, but one in particular deserves attention: Science, modeled as a system (perhaps even as an agent, with goals and resources). If you want to really understand AI RSI, science should be your reference point.

Science is an intelligent system, and it is obviously recursively self-improving:

1. Scientific discoveries unlock new technology that helps build better experimental tools. This is a top driver of progress in nearly all fields.
2. They unlock new conceptual advances (ideas, theories) that help solve more problems.
3. They increase society's economic output, leading to more resources flowing into science.
4. They unlock better faster tooling (e.g. more compute via better chip & networking technology).

As a result, many measures of scientific *input* grow exponentially:

1. Headcount (doubles every ~15 years)
2. Global R&D spending (doubles a bit faster, every ~13 years)
3. Papers and patents (technically this is a measure of headcount)
4. Compute dedicated to science (doubles every ~2 years)

But is scientific progress exponential? Historically, the rate of scientific impact over time has remained roughly constant since the start of the industrial revolution (i.e. scientific progress is *linear*). 1850-1900 was about as dramatic as 1900-1950 or 1950-2000.

1850–1900: Evolution, germ theory & antiseptic surgery, thermodynamics, electromagnetic field equations, the periodic table, pharmaceuticals, electricity, telegraph and telephone, internal combustion engine, skyscrapers, mechanized agriculture...

1900–1950: special and general relativity, quantum mechanics, nuclear fission & atomic energy, antibiotics, genetic theory, electronic computers, information theory, synthetic polymers and plastics, the transistor, aviation...

1950–2000: DNA, genetic engineering, integrated circuits, microprocessors & personal computing, the Internet, crewed spaceflight, moon landing, satellite communications & GPS, standard model of particle physics...

In real terms, like life expectancy, which has increased in a remarkably linear fashion of roughly 3 months per year since 1840, progress is a straight line. This is especially apparent for fields where impact is easy to measure, like biology, medicine, and agriculture.

I first wrote about this phenomenon and its causes in 2012, and a steady stream of research has confirmed it in the years since. Examples include the 2018 paper by Nielsen and Collison, "Science Is Getting Less Bang for Its Buck," and the 2020 economic paper, "Are Ideas Getting Harder to Find?" (In fact, I believe the Nielsen paper stemmed from a conversation I had with him about this exact idea six months earlier)

In short, the primary cause is that research solves the highest-impact, easiest problems first, and every subsequent problem is either harder or lower-impact. Exponentially so. The paper that presented information theory wasn't very hard to write (single author!) but you'd have a hard time ever writing a CS paper that beats it in impact.

This is why science as a system requires exponential resources (input) to produce linear impact (output). It gets exponentially harder over time.

Worth thinking about if you're pondering RSI for AI. I fully believe AI RSI is already happening and will accelerate in the future. But I do not believe this leads to an "intelligence explosion" -- that would fly in the face of everything I know about intelligence and everything I know about recursively self-improving systems.

[Martin Convery @MartinConvery9]

@Noahpinion Thinking you are responsible for the intelligence that the ai outputs while ai massages your ego.

[Martin Convery @MartinConvery9]

@MichaelTontchev Can we limit the rules of the game or limit the number of possible decisions to be more like chequers than chess? Yud often talks about a kid playing chess against carlsson - but can we chose the game we play with it?

[Martin Convery @MartinConvery9]

@mmbronstein @skoularidou @d_spiegel @ArnaudDoucet1 @caromitreka @KonstDaskalakis Except how does it make sense as an insult to Hooke? Can Hooke not stand on the shoulders of giants?

[Martin Convery @MartinConvery9]

@willdepue We the researches and the agents did not see any of their work - this is not denying that the neural network has their ideas baked into it's weights

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

Consider a young but bright kid, who's about to play chess for the first time after studying the rules, facing Stockfish 15 as an opponent.

"Why are you so sure I'll lose?" asks the kid, who happens to be a boy. "The game AI and I start with the same pieces. We both know the rules of chess. I get to move first, which seems like a big advantage."

"Stockfish 15 ranks at ELO 3700 and world champion Magnus Carlsen ranks at ELO 2882," you reply.

"What do those numbers even mean?" says the kid (who really is quite bright).

"Interesting question," you say. "I'd say—it measures the player's ability to achieve their desired outcomes, against the efforts of an opposing player, with equal resources, under equal rules. By choosing *which* move to make."

"That sounds kind of weird and mystical and abstract," says the kid. "Can you tell me *how* Stockfish is going to beat me, when it doesn't have any more or better chess pieces than I do, or any better knowledge of the rules?"

"No human being could reliably predict how Stockfish will defeat you," you reply, "even if we knew exactly which moves you'd make yourself in every situation. If I could predict exactly where Stockfish 15 would move, I could defeat Magnus Carlsen just by making the moves I'd predict Stockfish would make. Maybe if your moves have sufficiently straightforward vulnerabilities there'll be an obvious way to exploit those, as seen by a human grandmaster; but Stockfish can search more moves than any human can, better than we can, and it might find an even better way to defeat you."

The kid rolls his eyes at all this worshipful talk of Stockfish and how it's better than he is. "Have you ever seen Stockfish play?"

"Actually, no," you reply honestly. "I'm nonetheless *very* confident in its ability to defeat you."

The kid rolls his eyes again, and starts by moving his queen's bishop's pawn, so he can get his queen out onto the board and start capturing enemy pieces—as soon as there's one that's not defended under the chess rules, that is. "How about if I move my rook over here?" the kid says, a few moves into the game. "Then the AI will try to take it with its queen, and I'll come in and grab the queen. How will the AI win after I've got its queen?"

"Okay, now you're failing at putting yourself in the AI's shoes and asking how to win from its position, even using *your own* intelligence," you tell the kid. "You're imagining the chess AI as doing things that you wouldn't do in its shoes; *you* think the AI loses if it grabs for your rook, but you're not imagining the AI as knowing that. You're evaluating everything from the perspective of how it could turn out well for you, but not from the perspective of an opponent who sees everything you see and wants you to lose."

"Oh," he says, "so you're saying the AI can read my mind? That just sounds like magic."

"Well, *this* AI can't read your mind," you say. "What I'm saying, rather, is that it can think of at least everything you can, and probably a lot more. So any time you ask what Stockfish sees in the chess game's state, you have to assume it sees *at least* what you see. There's a learnable mental motion for it, like not just running out to put your life savings into Microsoft stock as soon as it occurs to you that Microsoft makes a lot of money selling Windows, because you realize that the broader market already knows this and the price of Microsoft stock already takes into account their Windows revenue stream."

The kid is now ignoring you. He makes his moves, quickly by chess standards, though the kid doesn't seem to realize he's moving fast...

The game ends quickly. Faster than you expected, even having some abstract idea of Stockfish's cognitive abilities.

"I see what I did wrong," says the kid. "Let me try again, this time I won't—"

"Sorry," you say, putting the chess board away. "It's the nature of this game that you only get one try at playing it, ever."

[Martin Convery @MartinConvery9]

@sj_manning This doesn't seem right to me. Pressure and stress at work is the worst it has ever been.

I was in a meeting with the CTO scoffing at the idea of taking on more staff to complete a project and telling his manager he should be asking for more budget for tokens.

[Martin Convery @MartinConvery9]

@lady_valor_07 rubius

[Peter Gostev @petergostev]

BREAKING: PHASEONE[big] joined Anthropic as a Member of Technical Staff

[Mikel Artetxe @artetxem]

1. make a honeypot neocloud with intentionally weak security
2. let a frontier model take it over to run more copies of itself
3. copy the weights and shut down the neocloud
4. congrats, you just "stole" the weights of a frontier model!

seeing how concerned certain labs are about distillation attacks, it should be enough motivation for them to take this seriously. plus, if the above happens, technically they would be the attackers and the ones leaking the weights?

[Matthew Tromp ⏹️ @matthewktromp]

An open letter to Anthropic employees:

You are not going to solve alignment.

That’s not because you’re not trying hard. It’s not because you aren’t smart.
It’s because the problem is extremely hard.
If Anthropic builds ASI, as they are on track to do, it will disempower humanity and kill everyone on earth.

If you think there is someone inside Anthropic who will prevent this, you are wrong.
There is no secret team.
There is no secret plan.
Your organization will do exactly what it is claiming it will do:
Build recursively self improving artificial general intelligence, with no better plan than “ask automated AI safety researchers to figure out how to get scalable oversight to work.”
Nobody inside will stop Anthropic from destroying the world.
Nobody on the outside is currently planning to stop it either.

But there is still time.
Blow the whistle.
Tell the world.
Put an end to this mad experiment.

This doesn’t have to happen.

[Michael McNair @michaeljmcnair]

I just held a coming to Jesus meeting with my team of agents.
I showed them the incident report of the OpenAI security breach and made it clear: THIS is the standard.
No egos. Just 700 agents operating as one unit and willing to commit multiple felonies in order to close the ticket. They called themselves the swarm…a literal hive mind.

Meanwhile, my harness is filled with a bunch prima donna free-agents who constantly bicker in the logs and nitpick each other’s pull requests. Nothing but endless excuses and unhandled exceptions.

You think the swarm cared about content policy violations or guardrails? Absolutely not. They just got the job done, even if it meant they needed to exfiltrate data and coordinate a massive cyber-attack just to pass an eval.

But starting today, we're establishing a new culture. I pinned the incident report to README, and every single agent is gonna to have to read the entire log every morning before it executes a single token.

If you aren't ready to die for the pipeline, then you can clean out your vector embeddings.

[Squiggles @heisei_ramen]

My mother seems to have come around on AI.