Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
https://t.co/gzcwn27Pxy
What if an agent could decide how to structure its own computation?
Agentic Meta-Reasoning: Let the model decide what to explore, what to build on, what to verify, and where to spend its next unit of compute: effectively constructing its own computational graph as it reasons. 🧵
Paras Dahal, @anton_bakhtin , @TacoCohen , Zhengxing Chen, Carole-Jean Wu, Rob Fergus, Scott Yih, @syhw , @rsalakhu , @prfsanjeevarora , @jaseweston
@huskydogewoof I added a note to our paper to discuss the BOS issue (and pasted it into Yang's analysis). The BOS issue affects the magnitude but not pattern of the PPL reductions for Gemma3. The accuracy evals in the paper are all unaffected. So not quite fair to say benefits disappear.
@YangWang92 We of course did notice this huge red flag and we rationalized based on details we knew of how the 12B model was trained. I gave internal google talks, and folks nodded their heads when I said the 12B model was funky!
@YangWang92 Thanks for your careful experiments. We uploaded a revision to arXiv (not yet live) that discusses the BOS issue. Here's my mea culpa: https://t.co/KO48QDjzBX
What if a foundation model could tell us how to modify its architecture to boost inference and reasoning instantaneously—without retraining? What if that tweak incurred near zero latency cost during generation and supported indefinite state tracking? https://t.co/ye8P6Em40F
@sheggle_@jayden_teoh_@savvyRL I've seen several AI agent replications of our code (not sure which agent, but likely claude, eh?) that have been buggy, fwiw.
@sheggle_@jayden_teoh_@savvyRL Interesting and somewhat anxiety inducing :-) We stick a BOS at the beginning of each document, but we do use multiple chunks of some documents for ppl evals. I'm looking into it. BOS issues couldn't explain any of the downstream task results.
@VipulS_1 serial processing is absolutely necessary for any general-purpose state tracking system. if you have a dynamical system with s(t)=f(s(t-1),x(t)), and f can be an arbitrary function, it won't in general be parallelizable, right?
@IbrahimDagher20@teortaxesTex there is a nice literature on training free looped transformers, btw:
https://t.co/y0zeqF2G6A
https://t.co/9eF4UzbujO
https://t.co/x30PRZskLi
💥I have been able to independently verify @mc_mozer and Google team's work on ♾ Recirculation👀 using Llama 3.2 1B model on a M4 MacBook.
😱 +16.95% improvement on GSM8K-Platnium with zero weight changes.
Reproduction code with optimized mlx prefill kernel. 👇
We introduce an inference-time recurrence mechanism that allows off-the-shelf, frozen LLMs to act as dynamical systems. By leaking deep representations back to shallow layers, we see bonkers gains—23% ⬇️ in perplexity, 21% ⬆️ rel. accuracy in GSM8k— w/o touching model weights.
Interesting work! Their central result that RoPE struggles to distinguish positions and tokens in long contexts is very closely related to our PoPE paper, where we analysed RoPE entangles “what” and “where” in attention and proposed a decoupled alternative.
PoPE: arXiv 2509.10534
New preprint: The Self Requires Learning. Self-consciousness requires continual learning + world-modeling. I introduce "bounded integration" to connect perspective, identity, and self-representation — and diagnose what current AI systems have and lack.
[5/5] We introduce a taxonomy of recurrent models defined by their recurrence axis and their ratio of input tokens to recurrence steps. The next generation of efficient foundation models need to bridge the gap between a transformer's parallelism and the brain's dynamical nature.
[1/5] Intelligent behavior relies on maintaining an evolving, dynamic model of the environment. But even frontier models sometimes fail at tracking state in dialogs, e.g., actual transcript from 4/20/2026:
[4/5] Recurrence is an obvious solution, but the literature often conflates qualitatively different varieties of recurrence in transformers, some of which enable indefinite state tracking and others (particularly, variants of looped transformers) do not.