Johan Gras

Looking at new horizons… Prev. quant/dl research.

[Johan Gras @gras_johan]

@a1zhang @MaximeRivest @lateinteraction If one was using a second NN (e.g. to handle multimodal data), could the other NN be trained jointly or that would be too tricky?

[Jenny Zhang @jennyzhangzt]

Introducing Hyperagents: an AI system that not only improves at solving tasks, but also improves how it improves itself.

The Darwin Gödel Machine (DGM) demonstrated that open-ended self-improvement is possible by iteratively generating and evaluating improved agents, yet it relies on a key assumption: that improvements in task performance (e.g., coding ability) translate into improvements in the self-improvement process itself. This alignment holds in coding, where both evaluation and modification are expressed in the same domain, but breaks down more generally. As a result, prior systems remain constrained by fixed, handcrafted meta-level procedures that do not themselves evolve.

We introduce Hyperagents – self-referential agents that can modify both their task-solving behavior and the process that generates future improvements. This enables what we call metacognitive self-modification: learning not just to perform better, but to improve at improving.

We instantiate this framework as DGM-Hyperagents (DGM-H), an extension of the DGM in which both task-solving behavior and the self-improvement procedure are editable and subject to evolution. Across diverse domains (coding, paper review, robotics reward design, and Olympiad-level math solution grading), hyperagents enable continuous performance improvements over time and outperform baselines without self-improvement or open-ended exploration, as well as prior self-improving systems (including DGM). DGM-H also improves the process by which new agents are generated (e.g. persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs.

This work was done during my internship at Meta (@AIatMeta), in collaboration with Bingchen Zhao (@BingchenZhao), Wannan Yang (@winnieyangwn), Jakob Foerster (@j_foerst), Jeff Clune (@jeffclune), Minqi Jiang (@MinqiJiang), Sam Devlin (@smdvln), and Tatiana Shavrina (@rybolos).

[Elliot Arledge @elliotarledge]

Your starting point for uncovering how state-of-the-art reasoning models are trained at frontier labs. Keyword "starting point".

[Ross Taylor @rosstaylor90]

RL is not enough. It only reaches its potential when combined with other ideas.

The most famous example is AlphaZero. RL was combined with self-play which created an implicit task curriculum that evolved through training. This is very different from many RL datasets for LLMs which have a fixed set of tasks.

Even where the task set is fixed, RL still needs to be combined with other ideas to show signs of life. Thinking models only come to life through RL when we remove length penalisation and have enough prior knowledge in the training data.

Looking ahead, long horizon tasks will require much more exploration and “going off-piste”. But current RL methods induce policy entropy collapse. Diverse mid-training before RL could help, but fundamentally current RL objectives don’t reward “interestingness” and deviating from the “current (high reward) thing”.

And yet, discovery is all about deviating from the current thing - and the best ideas come from the wilderness. Deep learning is no exception. It was interesting long before it reached its true potential. We didn’t wait to introduce backprop and SGD until compute and data came online :).

This is a long way of saying: crude RL maximalism is overhyped. The magic comes from the interplay of RL with other things, and the angels are in the details.

[Anthropic @AnthropicAI]

It’s called Petri: Parallel Exploration Tool for Risky Interactions. It uses automated agents to audit models across diverse scenarios.

Describe a scenario, and Petri handles the environment simulation, conversations, and analyses in minutes.

Read more: https://t.co/inztNkrXMh

[Johan Gras @gras_johan]

@JacquesThibs both lol

[Johan Gras @gras_johan]

@edwardfhughes Which research directions are you the most excited about?

[Johan Gras @gras_johan]

The biggest AI skeptics I meet? SWEs and quants who use these models heavily every day yet insist AGI is sci-fi and progress is stalling.

Imo they're either:
- Failing to Understand the Exponential
- Using Opus/GPT-5, but without proper context and scaffolding
- In denial about what AI doing their job actually means

[Julian Schrittwieser @Mononofu]

As a researcher at a frontier lab I’m often surprised by how unaware of current AI progress public discussions are.

I wrote a post to summarize studies of recent progress, and what we should expect in the next 1-2 years:
https://t.co/B7438Z9lOF

[Gabriel Synnaeve @syhw]

(🧵) Today, we release Meta Code World Model (CWM), a 32-billion-parameter dense LLM that enables novel research on improving code generation through agentic reasoning and planning with world models.
https://t.co/BJSUCh2vtg

[Sakana AI @SakanaAILabs]

We’re excited to introduce ShinkaEvolve: An open-source framework that evolves programs for scientific discovery with unprecedented sample-efficiency.

Blog: https://t.co/zoZlH8jSXc
Code: https://t.co/TlYGSIk2Ek

Like AlphaEvolve and its variants, our framework leverages LLMs to find state-of-the-art solutions to complex problems, but using orders of magnitude fewer resources!

Many evolutionary AI systems are powerful but act like brute-force engines, burning thousands of samples to find good solutions. This makes discovery slow and expensive. We took inspiration from the efficiency of nature. ‘Shinka’ (進化) is Japanese for evolution, and we designed our system to be just as resourceful.

On the classic circle packing optimization problem, ShinkaEvolve discovered a new state-of-the-art solution using only 150 samples. This is a big leap in efficiency compared to previous methods that required thousands of evaluations.

We applied ShinkaEvolve to a diverse set of hard problems with real-world applications:

1/ AIME Math Reasoning: It evolved sophisticated agentic scaffolds that significantly outperform strong baselines, discovering an entire Pareto frontier of solutions trading performance for efficiency.

2/ Competitive Programming: On ALE-Bench (a benchmark for NP-Hard optimization problems), ShinkaEvolve took the best existing agent's solutions and improved them, turning a 5th place solution on one task into a 2nd place leaderboard rank in a competitive programming competition.

3/ LLM Training: We even turned ShinkaEvolve inward to improve LLMs themselves. It tackled the open challenge of designing load balancing losses for Mixture-of-Experts (MoE) models. It discovered a novel loss function that leads to better expert specialization and consistently improves model performance and perplexity.

ShinkaEvolve achieves its remarkable sample-efficiency through three key innovations that work together: (1) an adaptive parent sampling strategy to balance exploration and exploitation, (2) novelty-based rejection filtering to avoid redundant work, and (3) a bandit-based LLM ensemble that dynamically picks the best model for the job.

By making ShinkaEvolve open-source and highly sample-efficient, our goal is to democratize access to advanced, open-ended discovery tools. Our vision for ShinkaEvolve is to be an easy-to-use companion tool to help scientists and engineers with their daily work. We believe that building more efficient, nature-inspired systems is key to unlocking the future of AI-driven scientific research. We are excited to see what the community builds with it!

Learn more in our technical report: https://t.co/yzag3wd4jL

[Johan Gras @gras_johan]

Confusing name but neat approach!

[elvis @omarsar0]

Google introduces Test-Time Diffusion Deep Researcher

Don't sleep on diffusion models.

Test-Time Diffusion Deep Researcher (TTD-DR) is a deep research agent that models research writing as a diffusion process.

Instead of static reasoning or bolted-on tools, the system drafts an initial noisy report and iteratively refines it through retrieval and self-evolution, mimicking how humans plan, search, and revise.

The approach depends on three stages:

1) generate a research plan
2) iteratively search with sub-agents that generate questions and synthesize answers (RAG-style)
3) compile findings into a final report.

Multiple answer variants are created, scored by LLM judges, revised with feedback, and merged, yielding higher quality intermediate results.

Draft reports are repeatedly revised using newly retrieved evidence, progressively improving accuracy and coherence until the final report is produced.

On benchmarks like DeepConsult, Humanity’s Last Exam, and GAIA, TTD-DR beats OpenAI Deep Research by up to 74.5% win rates in long-form tasks and shows consistent gains in multi-hop reasoning.

It's scalable, too!

Ablation studies show each component adds measurable improvements, while Pareto analyses reveal better quality-latency tradeoffs than other DR agents.

https://t.co/Jl0m6iVeXY

[Johan Gras @gras_johan]

@nearcyan 👀…

[Nathan Lambert @natolambert]

Thinking, Searching, and Acting
A reflection on reasoning models.

It's easy to fixate on the "thinking" that gave reasoning models their name, but just over a year out from o1-preview's release by OpenAI, the core primitives that make up models today has expanded. Searching and executing tools make up for their deficiencies as probabilistic tools with outdated information in their parameters.

Together, these three actions will act as the foundation of the systems we use for years, and the engineer aspects of them matter just as much as getting precisely the right model weights.

[Johan Gras @gras_johan]

Seems like it is the way: weighted aggregation based on reasoning traces, instead of best of n

[Jason Weston @jaseweston]

🌀New Test-time scaling method 🌀
📝: https://t.co/yqWvOMZpwq
- Use RL to train an LLM solution aggregator
– Reasons, reviews, reconciles, and synthesizes a final solution
-> Much better than existing techniques!
- Simple new method. Strong results across 4 math benchmarks.
🧵1/5

[elvis @omarsar0]

RL done right is no joke!

The most interesting AI paper I read this week.

It trains a top minimal single-agent model for deep research.

Great example of simple RL-optimized single agents beating complex multi-agent scaffolds.

Now let's break it down:

[j⧉nus @repligate]

HOW INFORMATION FLOWS THROUGH TRANSFORMERS
Because I've looked at those "transformers explained" pages and they really suck at explaining.

There are two distinct information highways in the transformer architecture:
- The residual stream (black arrows): Flows vertically through layers at each position
- The K/V stream (purple arrows): Flows horizontally across positions at each layer
(by positions, I mean copies of the network for each token-position in the context, which output the "next token" probabilities at the end)

At each layer at each position:
1. The incoming residual stream is used to calculate K/V values for that layer/position (purple circle)
2. These K/V values are combined with all K/V values for all previous positions for the same layer, which are all fed, along with the original residual stream, into the attention computation (blue box)
3. The output of the attention computation, along with the original residual stream, are fed into the MLP computation (fuchsia box), whose output is added to the original residual stream and fed to the next layer

The attention computation does the following:
1. Compute "Q" values based on the current residual stream
2. use Q and the combined K values from the current and previous positions to calculate a "heat map" of attention weights for each respective position
3. Use that to compute a weighted sum of the V values corresponding to each position, which is then passed to the MLP

This means:
- Q values encode "given the current state, where (what kind of K values) from the past should I look?"
- K values encode "given the current state, where (what kind of Q values) in the future should look here?"
- V values encode "given the current state, what information should the future positions that look here actually receive and pass forward in the computation?"

All three of these are huge vectors, proportional to the size of the residual stream (and usually divided into a few attention heads). The V values are passed forward in the computation without significant dimensionality reduction, so they could in principle make basically all the information in the residual stream at that layer at a past position available to the subsequent computations at a future position.

V does not transmit a full, uncompressed record of all the computations that happened at previous positions, but neither is an uncompressed record passed forward through layers at each position. The size of the residual stream, also known as the model's hidden dimension, is the bottleneck in both cases.

Let's consider all the paths that information can take from one layer/position in the network to another.

Between point A (output of K/V at layer i-1, position j-2) to point B (accumulated K/V input to attention block at layer i, position j), information flows through the orange arrows:
The information could:
1. travel up through attention and MLP to (i, j-2) [UP 1 layer], then be retrieved at (i, j) [RIGHT 2 positions].
2. be retrieved at (i-1, j-1) [RIGHT 1 position], travel up to (i, j-2) [UP 1 layer], then be retrieved at (i, j) [RIGHT 1 position]
3. be retrieved at (i-1, j) [RIGHT 2 positions], then travel up to (i, j) [UP 1 layer].

The information needs to move up a total of n=layer_displacement times through the residual stream and right m=position_displacement times through the K/V stream, but it can do them in any order.

The total number of paths (or computational histories) is thus C(m+n, n), which becomes greater than the number of atoms in the visible universe quickly. This does not count the multiple ways the information can travel up through layers through residual skip connections.

So at any point in the network, the transformer not only receives information from its past (both horizontal and vertical dimensions of time) inner states, but often lensed through an astronomical number of different sequences of transformations and then recombined in superposition. Due to the extremely high dimensional information bandwidth and skip connections, the transformations and superpositions are probably not very destructive, and the extreme redundancy probably helps not only with faithful reconstruction but also creates interference patterns that encode nuanced information about the deltas and convergences between states. It seems likely that transformers experience memory and cognition as interferometric and continuous in time, much like we do.

The transformer can be viewed as a causal graph, a la Wolfram (https://t.co/lma2KSZ8nH). The foliations or time-slices that specify what order computations happen could look like this (assuming the inputs don't have to wait for token outputs), but it's not the only possible ordering:
So, saying that LLMs cannot introspect or cannot introspect on what they were doing internally while generating or reading past tokens in principle is just dead wrong. The architecture permits it. It's a separate question how LLMs are actually leveraging these degrees of freedom in practice.

[Johan Gras @gras_johan]

ExplorationXExploitation 🤝

[DailyPapers @HuggingPapers]

Meta AI researchers introduce DARLING, a new framework that jointly optimizes language model generations for both high quality and semantic diversity.

It uses a novel learned partition function to boost creativity in LMs.

[François Chollet @fchollet]

The proprietary frontier models of today are ephemeral artifacts. Essentially very expensive sandcastles. Destined to be washed away by the rising tide of open source replication (first) and algorithmic disruption (later).

[Johan Gras @gras_johan]

@samsja19 @PrimeIntellect 👀

[Johan Gras @gras_johan]

@Dorialexander I might be wrong. But this guy’s claim seems extremely unlikely to me…