Guanghan Ning

Research @fleet_ai, formerly @ ByteDance-Seed (LLM)
· views my own

[Guanghan Ning @quietnning]

Personal news: After ByteDance Seed and a stint as an independent researcher, I'm joining @fleet_ai as a Member of Technical Staff on the research team 🚀

Building Witness-inspired puzzle environments for ARC-AGI-3 convinced me that RL environments are one of the most under-explored bottlenecks on the path to truly capable agents. The team, the technical vision, and the open problems are a perfect fit. It’s exactly where I want to spend my next chapter.

If you're interested in building and scaling dynamic, complex environments that push the boundaries of agentic reasoning, would love to connect.

[Guanghan Ning @quietnning]

Enjoyed ICML week in Seoul with old and new friends. Also, of all 168 orals, 88 touch RL or environments. My take on the trends:

(1) RL envs and reward verification are the dominant hotspots right now, exactly the battlefield I care about most. The "env factory" paradigm, i.e., reliable env scaling, seems to be arriving (e.g., CVE-Factory: auto-conversion of real-world data at scale → executable, verifiable envs → 30%+ training gains; loop closed).
(2) tao2-Bench demonstrates programmatic, compositional task generation (that is verifiable), from the evaluation side.
(3) A trend in reward design: static verifiers → dynamic games. e.g., co-evolving rubrics (RLER), adversarial critique-based grading (Benchmarking at the Edge), and tool-grounded self-reward (Agent0-VL).
(4) CausalGame evaluates the causal-thinking capabilities of LLM agents through interactive games. ARC-AGI-3 has started a trend of using interactive gaming envs to evaluate specific LLM capabilities.
(5) Reward hacking has gone from accident to systematic research. Oversight, too, has gone from an engineering practice to a measurable science; monitorability now has metrics and scaling laws.
(6) Mid-training is now much better understood: "Midtraining Bridges Pretraining and Posttraining Distributions" shows how to choose mid-training proportions and, more importantly, timing: heuristics → concrete, applicable theory.
daVinci-Dev makes agent-native mid-training a cost-saving layer upstream of RL: instill agentic behavior via mid-training first, then apply RL. An open question worth quantifying: before running expensive RL, how much can mid-training on trajectories collected from the same environments cut the RL steps / env interactions needed to hit a target resolution rate?

[Nicolai Ouporov @ ICML @nicoup]

We do much more than sell training-data! And we are much much farther ahead than the numbers here show

Happy to see us recognized. Towards being the most profitable neolab! More soon

[Guanghan Ning @quietnning]

@davidsharff Thanks for your interest! Paper shall be coming soon, please stay tuned:)

[Guanghan Ning @quietnning]

Fable-5 finally revived, and I couldn't wait to run it on my benchmark (unofficial ARC-AGI-3 DLC. Official flattens everyone under 2%, this one actually separates them).

Quick numbers in the figure, take them with a grain of salt (4 seeds each).

Kind of underwhelmed though; I thought it'd clear Opus-4.8 on abstract reasoning by a wide margin, but it only edges it (44.7 vs 40.2). And yeah, RHAE is not the most stable metric I've ever seen in my life. 😂

Hoping to release the benchmark soon to aid the community's research on interactive rule/goal discovery and efficient execution.

[Guanghan Ning @quietnning]

Tufa Labs won ARC-AGI-3 Milestone 1 with the Duck, and here is my take:

The Duck is a pure externalized harness. A frozen, open-source Qwen3.6-27B in FP8 (running offline on one RTX Pro 6000 under a 9-hour cap for official Kaggle submission), driving a minimal Python REPL that inspects game state and acts. Zero training at eval time, no game-specific logic, one generic prompt across all 110 games. It's the successor to their own trained CNN specialist from last year; but they’re abandoning this trained-specialist lineage for a frozen model in a loop.

I had blogged about the spectrum of where the loop could live (tokens / harness / weights), and Duck lands on the harness end. It's actually the opposite from my bet about what Fable-5 probably does (loop into the weights).

Tufa's observations: the model reads a fresh board as a maze or an Atari-style game (mislead by pre-train data / cursed prior), hallucinates game sprites like robots, takes the energy bar for the goal. That's the wrong-level-of-abstraction failure I kept talking about. 1.21% is still on the floor; a scaffold can't patch it, because the abstraction forms in the model, not the REPL.

In the MLST interview, Jeroen pushes back on 'you shouldn't need a harness'(humans learned these patterns by playing games) and suggests RL gameplay training on today's best models (scoring ~1%) might beat a hardcoded harness. I think the team is independently pointing at the internalize direction for what's next. I am with them.

Huge respect to Tufa for open-sourcing it and inviting the community to collaborate. This is the strongest data point we have for the externalized side, and it shows how far that side reaches alone.

[Tufalabs @tufalabs]

Our winning solution to the ARC-AGI-3 Milestone 1 is now open-source: the Duck harness, built on Qwen 3.6 27B 🦆🔥

ARC-AGI-3 is interactive with no rules explained, the agent has to figure out the goal itself. We hit 1.21% with our lightweight harness.

Links below 👇

[Guanghan Ning @quietnning]

looking forward to their write-ups

[Greg Kamradt @GregKamradt]

.@tufalabs just open sourced their 1st place notebook 👀

[Guanghan Ning @quietnning]

Enlightening paper. The world model simulated with language is quite a natural way to represent various kinds of states. LWM can be decoupled from the LLM within an agent, which is straightforward; or it can be internalized and unified with the LLM, where world modeling can be guided (via system prompt) to take place in CoT.

With this paradigm, the hard part for ARC-AGI-3 is perception and representing the states at the appropriate abstraction level, in a robust way.

[Qwen @Alibaba_Qwen]

📣📣 Meet Qwen-AgentWorld — a native language world model that simulates 7 agent environments (MCP, Search, Terminal, SWE, Web, OS, Android) within a single model. Environment modeling is the training objective from day one, not a post-hoc adaptation.

🤔 LLMs are trained to be better agents — better at acting in environments. But nobody has trained them to model the environments themselves.

🗺️ Our roadmap: investigate how language world modeling can push the boundaries of general agent capabilities, along two routes:

1️⃣ Build a foundation model for environment simulation — outperforming Claude Opus 4.8 and GPT-5.4 on AgentWorldBench

2️⃣ Investigate how world modeling enhances agent training:
🔬 Controllable Sim RL (agentic RL with LWM as environments) surpasses training in real environments
🧠 Learning to predict environments (LWM warm-up) makes agents stronger — remarkably, even without any agent-specific training, this predictive knowledge transfers to agentic tasks with zero fine-tuning

📑 Paper: https://t.co/Jx2l5RKq71
📖 Blog: https://t.co/7tVcKyhsx2
💻 GitHub: https://t.co/B5Lvb1UZCn
🤗 HuggingFace: https://t.co/Kw3QBL1TM5
🧩 ModelScope: https://t.co/YBnGYgMWWI

[Guanghan Ning @quietnning]

@jaeahn24 thanks for reading;)

[Guanghan Ning @quietnning]

quick update on the Fable 5 / ARC-AGI-3 bet from last week. been collecting Fable 5 traces on arc-witness-envs since.

same env, same level. two abstraction altitudes.

opus 4.7's discovered rules sit at the surface: "agent navigates a grid; reach the target cell", and when that doesn't win, button-mash ACTION5 hoping it's a win trigger. trivial, layout-bound, doesn't transfer.

fable 5 names the actual mechanism: "the path partitions cells into regions; each star shares its region with exactly one same-color partner." that's the altitude where ARC rules actually live.

mechanism-level rules ARE the game. the mechanism-true traces go into the policy. the rest is noise.

full argument of what belongs in weights vs harness, and why HRM-style internalized loops matter for ARC-AGI-3: https://t.co/b0mPlvf0RW

[Guanghan Ning @quietnning]

Excited to see how Fable 5 performs on ARC-AGI-3, and here is my take:

1) The capability jumps on agentic benchmarks often come from the external shell, e.g., a self-referential, code-as-harness scaffold (Darwin Gödel Machine: freeze the model, evolve the scaffolding). However, it adapts via a slow analyze→rewrite→re-analyze loop.

2) ARC-AGI-3 is turn-based, so wall-clock isn't the constraint, interaction efficiency is (it scores actions or num of steps, not time). When sparse interactions demand a dramatic change of approach, an external rewrite loop is costly: every harness change we test spends scored steps (reset + retry). So I suspect the adaptive loop wants to live in the weights.

3) One architecture that internalizes that loop: a looped / recurrent-depth transformer. iterate shared weights in latent space = adapt computation depth on the fly. A Gödel-machine loop moved from code into weights. (Community reconstructions like OpenMythos speculate recent Claude models lean this way. Explicitly unconfirmed, not from any leak.)

If that bet is right, it should show up on fluid-intelligence tasks. Genuinely looking forward to Fable 5's ARC-AGI-3 number.

[Guanghan Ning @quietnning]

@mhmazur Since the recurrent depth architecture is itself a guess for Fable 5, we can't be certain no matter what the outcome is. It will remain speculative, but a jump would push me to go ablate this direction. I guess the real tell would be gains that scale with compute-per-token.

[Guanghan Ning @quietnning]

Excited to see how Fable 5 performs on ARC-AGI-3, and here is my take:

1) The capability jumps on agentic benchmarks often come from the external shell, e.g., a self-referential, code-as-harness scaffold (Darwin Gödel Machine: freeze the model, evolve the scaffolding). However, it adapts via a slow analyze→rewrite→re-analyze loop.

2) ARC-AGI-3 is turn-based, so wall-clock isn't the constraint, interaction efficiency is (it scores actions or num of steps, not time). When sparse interactions demand a dramatic change of approach, an external rewrite loop is costly: every harness change we test spends scored steps (reset + retry). So I suspect the adaptive loop wants to live in the weights.

3) One architecture that internalizes that loop: a looped / recurrent-depth transformer. iterate shared weights in latent space = adapt computation depth on the fly. A Gödel-machine loop moved from code into weights. (Community reconstructions like OpenMythos speculate recent Claude models lean this way. Explicitly unconfirmed, not from any leak.)

If that bet is right, it should show up on fluid-intelligence tasks. Genuinely looking forward to Fable 5's ARC-AGI-3 number.

[Greg Kamradt @GregKamradt]

We had access to Fable over the past few days

We were able to run it against public data but couldn't do semi-private (our private verification set) due to the new data retention policies

We're working with them to figure out a way to keep verification data private to ensure that ARC-AGI benchmarks continue to give us signal

fwiw, it did well on public data - we'll share once we're able to run semi private

[Guanghan Ning @quietnning]

Curious framing: wonder if the planning-depth limit is downstream of abstraction, not a separate failure mode.

Looking at this side-by-side: Opus 4.8 reasons cell-by-cell ("UP (4,6): empty. Safe."), while GPT-5.5 reasons in regions ("keeps access to the larger open right-side space," "unwinding along my tail-side corridor").

If the right abstraction compresses 5 tactical moves into 1 strategic choice, "5-6 step planning" becomes "1-2 step planning." Seems the bottleneck isn't horizon but representation.

Flagged this on ARC-AGI-3 last month too. Different game with similar upstream cause.

[Greg Kamradt @GregKamradt]

Opus 4.8 sitting at 7th on SnakeBench

Lost its final game against GPT-5.5

Models all still have the same failure mode.

They can plan 1-2 steps in advance, but moves that require 5-6 step planning (like 4.8's loss) are still out of reach

[Guanghan Ning @quietnning]

@GregKamradt 404 error, is the repo made public already?

[Nicolai Ouporov @ ICML @nicoup]

got to share how we build best-in-class dev tooling for our team at fleet! thx @ClaudeDevs

[Henry Lu @HenryL_AI]

🚀 Major update: We let the model design a multi-agent system — and it improved on ARC-AGI.

Self-evolution isn’t just the model improving itself. It’s the model acting as a researcher that designs other agents.

For single-agent tasks (coding, MCP-Atlas, etc.), we’ve shown the model can successfully design and evolve its own harness. But for harder tasks like @arcprize, the optimal solver is a complex multi-agent system.

The real question: Can the model evolve a multi-agent system from scratch?

We ran A-Evolve on ARC-AGI and proved yes — the model successfully evolved a multi-agent solver and achieved a clear performance uplift: 10% → 12%.

(Attached: one example task solutions of agent before & after A-Evolve)

Full results + details in the reply below 👇

#AgenticAI #AEvolve #SelfImprovingAgents #ARCAGI

[Guanghan Ning @quietnning]

Frozen Opus 4.7 (via OpenRouter), no RL post-training. "agent" here = scaffolding around a frozen model, not a trained policy. RL post-training is an option for open-weights models, but that's a separate workstream from this benchmark.
The stack: 4-stage FSM (recon → rule synthesis → planning → crystallization) with memory + rule-discovery modules. "Witness" in the name refers to the perception/decision layer being built for Witness-style envs, not RL-trained on them.

[Guanghan Ning @quietnning]

A data point alongside this:

Same model (Claude Opus 4.7), same env interface (ARC's official EnvironmentWrapper). Only the agent stack changes:
• my arc-witness-agent → 44 levels solved (36.7% on first 10 levels per game) on arc-witness-envs
• my arc-witness-agent → 0 levels solved (n=2 runs) on ARC-AGI-3
• ARC's reference agent → 0.18% on ARC-AGI-3 (per their blog)

Two independently-designed agent stacks, same collapse.

[Greg Kamradt @GregKamradt]

We just posted scores for GPT-5.5 and Opus 4.7 on ARC-AGI-3

Neither model made material progress, but the more interesting story is about *why* they didn't make progress

We reviewed every session they played to find common failure modes and studied what this tells is about real world tasks

There were three that surfaced:

- True Local Effect, False World Model - The models understand which action produced a change, but they fail to translate the effect into a global rule

- Wrong Level of Abstraction From Training Data - The models mistake an ARC-AGI-3 environment for another game

- Solved The Level, Didn’t Learn The Game - Even if a model beat a level, it’s unable to use that reward signal to enforce the correct actions

This analysis is important because aggregate scores (like most other benchmarks) mask a models thought process.

[Guanghan Ning @quietnning]

My agent has general components (memory, rule discovery, drawing from recent papers + OpenClaw) designed to be domain-general, but the perception/decision layer is still Witness-specific.
That's the more interesting finding: even scaffolding designed to be domain-general inherits domain-specific perception once shipped.
"Wrong level of abstraction" isn't just a model failure mode, it propagates through every layer humans add on top.
Caveats: single seed, 5k actions/game budget.

[Guanghan Ning @quietnning]

Great point! this decomposition is exactly what makes witness-style envs interesting. arc-witness-envs tries to separate the two by varying rule complexity while holding spatial priors constant; if a model solves simple-rule puzzles but fails on complex-rule ones in the same layout, that points to rule formation rather than exploration. but yeah, doesn't fully isolate them. adding a state-space coverage metric (unique states explored / branching factor) alongside success rate seems like the obvious next step. currently this exists in my agent harness but not in the envs. seen any good ablation designs for this? would love to learn from prior work.

[Guanghan Ning @quietnning]

Personal news: After ByteDance Seed and a stint as an independent researcher, I'm joining @fleet_ai as a Member of Technical Staff on the research team 🚀

Building Witness-inspired puzzle environments for ARC-AGI-3 convinced me that RL environments are one of the most under-explored bottlenecks on the path to truly capable agents. The team, the technical vision, and the open problems are a perfect fit. It’s exactly where I want to spend my next chapter.

If you're interested in building and scaling dynamic, complex environments that push the boundaries of agentic reasoning, would love to connect.

[Guanghan Ning @quietnning]

@fleet_ai huge thanks to @nicoup and @andrewthezhou for the welcome package and the trust 🙏