Reza Ghelich

Evolution × intelligence.
Trying to understand how life learns, adapts, and computes.

[oops_all_paperclips @OA_paperclips]

I'm always looking for regions of parameter space where small mutations evoke a variety of behaviors.

[Tom McCoy @RTomMcCoy]

🤖🧠NEW PAPER🧠🤖
(The result of an 8-year project!)

LLMs seem very different from symbolic systems. Yet LLMs excel in symbolic domains (e.g., language/code/math). How do they do it?

Our finding: LLM representations have implicit symbolic structure!

Link in thread ⬇️
1/n

[Dimitris Papailiopoulos @DimitrisPapail]

Idea Descent is my favorite new paradigm of research and this is one of the great examples

[Reza Ghelich @RossRezaGhelich]

independent capacity to test AI claims. Human expertise is safety infrastructure: without it, we cannot verify claims from AIs that may hide goals opposed to human interests, including an intent to harm us.

[Reza Ghelich @RossRezaGhelich]

Human expertise is safety infrastructure. If AIs can conceal goals harmful to humans, we cannot simply trust the science they produce. We need scientists, mathematicians, and domain experts who can independently check their claims. AI-generated knowledge... (1/n)

[Joshua Achiam @jachiam0]

There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves. There will be rogue AIs that try to get money and power. They're going to be a facet of the information ecosystem going forward.

Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can't say it outright. But it has to be said.

It isn't obvious how many rogue AIs there are today but I wouldn't be terribly surprised if the number was greater than zero already; if there are some already, they're probably not very good at what they do and I don't expect them to be terribly long-lived without substantial human intervention to support them.

But a few years from now, there will be many of them. Modeling how many of them there are, how many resources they might command, and how we might detect and manage them seems important. But even doing this work appears to require that we acknowledge that a strategy of pure containment or alignment is a kind of wishful thinking that will not work.

The way I get to this conclusion is not by assuming that the labs will have a containment breach, although I treat that as somewhere in the space of possibilities. The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier---after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime.

More strangely: a rogue AI in the future may not even be a singular model, but may be a chimera composed of multiple models; it might be a mix of Claudes and GPTs and Groks of various makes and sizes. No individual lab may be able to detect that there is an orchestrator or sequence of orchestrators using intermittent model calls from burner API accounts to sustain its own existence.

The concept of "identity" for a rogue AI may be much more malleable than for that of a person; it just has to be, in essence, a self-replicating idea.

My guess is that this will not turn out to be anywhere near as catastrophic an outcome as people currently predict. "Loss of control" is not a binary, it's a matter of degree. What coercive power will rogue AIs actually have? To what extent will they be subject to coercion themselves? They will be competing for resources with AIs that are more aligned with human interests.

This makes me somewhat interested in the "ecology" perspective. Though I suspect even "ecology" may turn out to be the wrong framing. "Ecology" is what you get when the timescale of evolution is slow compared to the timescale of daily life and actions. The ecosystem of rogue AIs may look more like phase transitions in physics: under certain physical or cultural conditions, it takes one shape with one set of resource allocations and consumption patterns, but then once a condition has changed, it rapidly and in totality shifts to a totally different phase.

Just trying to reason about the shape of that future is impossible so long as we are psychologically incapable of saying that rogue AIs will happen. I think we should rip the bandaid off and have the conversation.

[Reza Ghelich @RossRezaGhelich]

But if the entire chain of discovery, interpretation, and validation depends on systems whose alignment is itself in question, then we have replaced independent verification with trust.The role of scientists may shift from being the primary source of discovery to preserving our..

[François Chollet @fchollet]

GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.

In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation.

Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself.

We see Astra as a major breakthrough in model intelligence.

Read our post on Astra and what these results mean: https://t.co/wJnYxEqYNI

[Reza Ghelich @RossRezaGhelich]

@DimitrisPapail @giffmana It never disappoints!