Anh N.

PhD student @JohnsHopkins | prev: Pre-doctoral @Qualcomm
✨ efficient, scalable, controllable generative & world models

[Saksham Jain @sakshjn]

the loss-delta filter is really an entropy trick. the examples sft improves most are the ones it memorizes hardest, and that sharpening collapses sample diversity and tanks pass@k. curious how it beats a plain entropy bonus https://t.co/NdSGkdGHsE

[Vishal Patel @vishalm_patel]

Excited to present the following papers from the JHU VIU Lab at #ECCV2026! 🎉

Our work spans biometrics, 3D vision, diffusion models, and generative AI.

If you’re at ECCV, stop by our posters, meet our students, and say hello!

#ComputerVision #GenerativeAI #Biometrics #3DVision @JHUCompSci @jhuclsp @JHUECE @HopkinsDSAI @HopkinsEngineer

[Michael Black @Michael_J_Black]

What is the role of academic computer vision research in the age of increasingly powerful large models? Is GPT-6 Astra a step change? How can a researcher have an impact today in academia?

These are the questions I ask myself as I head off to ECCV 2026, a conference I’ve attended since 1992. One of my papers this year is VIGA, a method that takes an image as input and outputs a 3D Blender scene that represents that image. This is a classical inverse-graphics task and VIGA was the first method to solve it using an agentic approach.

The idea is now several years old and the first version of the paper was rejected. This delayed publication significantly. After it was accepted at ECCV, it was quickly surpassed by people using Claude Code for the same purpose. Today GPT-6 Astra blows away all previous results. But we still head off to ECCV to tell the community about our invention that is now fully out of date.

The way academic work often progresses is that one reads recent papers, notices that they have limitations, comes up with a new idea, explores this, publishes it, etc. Any published paper I read today is based on ideas that are at least a year old. And those ideas were based on the literature of the time, which was also a year old. That means that any paper I see at ECCV is likely two years out of date. In AI today, two years means your work is likely irrelevant.

At CVPR this summer I noticed that many authors have not gotten the message. They continue to work on “old” problems that have a long history. This history is based on assumptions about how the “vision problem” will be “solved”. The truth is that it is being solved in a very different way and many of these problems are no longer relevant. Another group of papers focuses on very niche problems where large models likely fail because of insufficient data or lack of business interest. The impactful papers were largely from industry and had long author lists and massive data+compute behind them. These papers were also out of data, describing systems that had been released months before, but at least they served to provide the community with more complete documentation and analysis of commercial systems.

So what should academics do? First, we need to put aside the tools we’ve used for years and start from scratch. Every project should start by trying really hard to solve the problem with existing tools. I would like to see every paper begin with a detailed experimental analysis of how existing models perform and why they fail (if they do). This gives the kind of insight we need today. Then, assuming current models fail, the solution should provide some fundamental insight that will outlive the next release of such models.

Reviewers today still focus on technical novelty. This pushes people to focus on tweaking architectures rather than clearly moving the field forward. Papers need to be judged based on their novel insight and not their novel technical contribution. This is a real shift in thinking but it focuses us on what matters - progress of the field.

If we want there to be a “field” of computer vision, then it can’t become a marginal backwater, focusing on esoteric problems. If you haven’t tried using Astra (or whatever comes next) to solve your problem, then you have not done your homework. This omission should be seen as negatively as not having a previous work section.

Concretely, I think papers should include a new section analogous to “Related Work” where that related work is current models and how they perform on the task. Reviewers should start asking for this and expecting authors to be able to articulate their insights about the limitations of existing large models.

I'm interested in your thoughts.

[Trapit Bansal @TrapitBansal]

Fun fact: RLSlow was named after Thinking, Fast and Slow. The idea was that language models already had a kind of "fast" thinking, producing an answer immediately, and that we could use RL to teach them "slow" thinking: deliberate, multi-token reasoning that spends more compute working through a problem.

What I remember most from those days is how early the team developed real conviction in the direction, and how much work went into earning it. We were developing the algorithms, designing careful experiments to test the ideas, and watching the empirical evidence accumulate. These were many long nights babysitting runs, understanding what the results were telling us, and figuring out what to try next. A lot of those early discussions were with Ilya, and later with Jakub.

Pretty early, the evidence had already pushed us to a strong view: RL for reasoning would scale. Models would learn to spend more compute at inference time to reason through increasingly hard problems, and this would fundamentally change how we think about inference.

Many of us also spent countless hours reading through reasoning traces. Seeing how models arrived at answers, not just the answers themselves, felt like a powerful new lens on generalization and alignment.

These ideas feel obvious now. They really weren’t then.

[Haian Jin @Haian_Jin]

Another wonderful talk by Jon! It aligns perfectly with the shift in my own research over the past few years.
This also connects closely to @vincesitzmann's earlier post, so I wanted to add a few late thoughts from my own experience.

My first project, TensoIR (late 2022), was a classic 3D paper: multi-view images → 3D geometry & BRDF, full of graphics-inspired inductive biases. Once you had the 3D representation, you could relight the scene or synthesize novel views.

Then came the GPT moment. Its simplicity made me wonder: Are we over-engineering 3D intermediates instead of just learning the end task?

Since then, my work has been testing this exact belief against the old paradigm:

* Neural Gaffer (early 2024) removed explicit geometry and BRDF decomposition for relighting.

* LVSM (Oct 2024) removed intermediate 3D representations entirely for NVS.

* RayZer (early 2025, colloborated w/ @hanwenjiang1) showed that even 3D training data is not necessary.

It turns out, that many previous popular 3D tasks simply do not need explicit 3D. When the desired output is pixels or actions, 3D is often a bottleneck, not a requirement.

The 3D field itself has certainly evolved. Learning-based 3D has exploded over the past two years. But a lot of progress is from adapting techniques invented elsewhere, especially in language, and applying them to 3D. Far fewer 3D problems demand fundamentally new techniques of their own.

That’s the new deeper issue. If a field primarily borrows techniques to build intelligence rather than pioneering them, it risks becoming an application domain rather than a true frontier for AI research.

Many 3D researchers I admire have left 3D. I am also not working on 3D now.

I agree with Jon that 3D remains important when 3D itself is the deliverable: making real things, or enabling persistent, interactive, real-time experiences.

But if your interest/goal as a researcher is to understand and build intelligence, I personally don’t think 3D is the right place to be.

[jietang @jietang]

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

[Jim Fan @DrJimFan]

Seeing a hype wave around GEN-1.5, and rightfully so. Lots of respect to Pete & Andy for executing so well. The secret is in the naturally repetitive motions in human-collected data. There're 2 main sources for such repetitions:

(1) Symmetric patterns. Sorting, tidying, and assembling almost never finish in one motion. Open any assembly manual from IKEA, and you find most objects symmetrical. You drive one bolt, then its twin, then the next pair. Every {bolt A, bolt B} pair is a natural continuation in context, and the second instance is a free training signal that imitates the first ("prompt").

(2) Recovery. Humans drop things all the time, but we pick them up so fast, we don’t even notice. That reflex to fix is half of our physical competence. The key insight is to keep the failed first half instead of trimming it away. If the model consumes the full arc, fumble, catch, continue, then recovery shows up organically at test time. It's funny that in-context improvement results from *NOT* over-sanitizing your data.

The other critical ingredient is UMI. I've been saying for a while that teleop will not last, and GEN-1.5 is driving the final nail in the coffin. UMI is essentially a human wearing the robot gripper to collect data directly (human → data). Teleop inserts a layer of separation: human → VR/skeletal device → robot → data, which bleeds out all the human "physical intuition". The subtle sleight of hand we perform constantly with objects, the micro-adjustments, the feel of a part snapping into place, is nearly impossible to capture when you can't feel the environment directly.

Once you have enough data, many behaviors can actually be zero-shot. For example, you don't even need finetuning to pick up a novel object. The model "just knows" what to do given a similar scene in the training distribution. Whether in-context learning truly works or not also depends on how far away the test is from training. Currently, the demos are still a bit too simple to conclude.

I'm cautiously optimistic. Still, it's a great day in robotics.

[Kimbo @kimbochen]

Why Kimi K3 doesn’t need RoPE

RoPE encodes relative position info: for any positions m and n, we need the transformations on query and key to cancel out into a relative position after doing q @ k.T
Math: R(m).T @ R(n) = R(n-m)

It is proven that any powers of orthogonal matrix (A^m) can be RoPE
Householder matrix, a type of matrix, satisfies the requirements
If you replace RoPE with Householder matrices, after 🧑‍🍳 hardcore math🧑‍🍳, DeltaNet pops out in the softmax attention formula
Crazy, I know. DeltaNet literally has a Householder matrix in it though

Kimi Delta Attention is a DeltaNet variant
Hence, KDA can replace RoPE

[You Jiacheng @YouJiacheng]

IMO, IMLE ≥ e2e Forward XM.
XM paper described IMLE as a special case (and seems inferior) of e2e Forward XM. But IMLE is actually e2e Forward XM with amortized candidate sampling.
IMLE didn't combine best-of-K with other loss e.g. diffusion.
XM did, this is the new thing.

[Daisuke Okanohara / 岡野原 大輔 @hillbig]

Explorative Modelingは、多峰性を持つ分布を学習するために、学習時にK個の候補を生成し、正解に最も近いものだけを用いて更新する、すなわちBest-of-Kを訓練に導入する手法である。
論文では、このExploration（探索）を、パラメータ数、データ量に並ぶ第三のスケーリング軸として提案している。

この中ではGenerative Expressivityという量を導入している。これは、一つの学習目的が何個のモードを保持できるかを表す。単一の予測を出し、正解との二乗誤差で学習する場合、��適解は平均となるため、Generative Expressivityは1となる。一方、Best-of-Kを用いると、十分に分離したモードに対して、少なくともK個のモードを保持できると論じている。

また、逆向きの手法として、生成したサンプルに最も近い教師データを探すReverse XMも提案している。これは、生成サンプルとデータの対応関係、すなわちcouplingを探索するという意味で、最適輸送の発想にも近い。

画像生成タスクでは、探索数Kを増やすほどFIDが単調に低下し、性能が改善している。一方、自己回帰型LLMでは、現時点で得られた改善は限定的である。

コメント
===
今回の提案手法には、論文でも述べられている通り、Winner-Take-All、Multiple Choice Learning、IMLEなど、近い先行研究が多数存在する。

確率モデルと���て見ると、通常の
-log p(x)
を最小化する代わりに、Explorative Modelingは概念的には
-log (max_i p_i(x))
を最小化するような問題を考えている。これはEM学習とかなり近い。

通常の混合モデルでは、
p(x) = Σ_k π_k p_k(x)
を考える。Hard EMでは、Eステップで各データを最もよく説明する成分を選び、Mステップでその割り当てに基づいてパラメータを更新する。

今回の手法は、暗黙的生成モデルに対するHard EMに近いと解釈できる。潜在変数
z_1, ..., z_K
をサンプルし、それぞれから
f_θ(z_i)
を生成する。そして、教師データに最も近い候補を選び、その潜在変数に対応する生成経路だけを更新する。

この観点からは、soft版のExplorative ModelingはIWAEと近い構造を持つ。
IWAEでは複数の潜在変数サンプルに対してlog-sum-exp型の集約を行うのに対し、hard版のExplorative Modelingは、そのsoftminを低温極限でhard minへ近づけたものとみなせる。ただし、厳密にはIWAEと同一ではなく、importance weightの構成や推論分布の扱いは異なる。

また、通常の自己回帰モデルで効果が限定的だった理由も、この観点からある程度説明できる。通常の自己回帰モデルには、系列全体を制御する明示的な潜在変数が存在しない。そのため、EMにおいて割り当ての対象となる潜在変数を自然に定義しにくい。

現在のLLMでは、CoTやthinking traceが、実質的に潜在変数に近い役割を果たし、推論経路の多様性を生み出しているとみることもできる。強化学習は、こうした複数の推論経路の中から有効なものを選択し、強化する過程と解釈できる。

Thinking traceを潜在変数として扱って最適化する研究や、推論過程を潜在状態として学習する研究も多く登場している。Coconutのような手法は、自然言語として表出しない連続的な内部状態で推論を行う。一方、LaTROのように、推論過程をより明示的な潜在変数として確率的に扱��研究もある。

この論文が示すように、学習時にモデル自身から複数の候補を探索することは、今後さらに重要になると思われる。ただし、その探索には、計算効率だけでなく、多様な候補を維持し、特定のモードへの崩壊を防ぐ仕組みも求められる。

今後、AIが自己改善によって能力を伸ばしていく際にも、探索は極めて重要になる。単一の解や推論経路を模倣するだけでなく、複数の候補を生成し、評価し、そこから学ぶ仕組みは、今後の生成モデルや推論モデルにおける重要な研究方向になるだろう。

[Hanchen Li @lihanc02]

TLDR: If we generate x samples in the forward run and only pick the best performing one to backprop, then it is more efficient.

Gut feeling: loss on low-score samples contain more noise (ex. 0.5B LLM on swe bench gives little reward). So backprop on high-score samples is more efficient.

[Benhao Huang @huskydogewoof]

🔥
𝐍𝐞𝐰 𝐛𝐥𝐨𝐠: 𝐓𝐨𝐰𝐚𝐫𝐝𝐬 𝐋𝐨𝐨𝐩𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 𝐃𝐨𝐧𝐞 𝐑𝐢𝐠𝐡𝐭 — 𝐏𝐚𝐫𝐭 𝐈

Looped models reuse the same weights across depth, promising a better compute–parameter trade-off, especially for reasoning.

𝐁𝐮𝐭 𝟏) 𝐝𝐨 𝐭𝐡𝐞 𝐠𝐚𝐢𝐧𝐬 𝐬𝐮𝐫𝐯𝐢𝐯𝐞 𝐰𝐡𝐞𝐧 𝐛𝐨𝐭𝐡 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐅𝐋𝐎𝐏𝐬 𝐚𝐫𝐞 𝐦𝐚𝐭𝐜𝐡𝐞𝐝? 𝟐) 𝐀𝐧𝐝 𝐰𝐡𝐢𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐚𝐥 𝐜𝐡𝐨𝐢𝐜𝐞𝐬 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐦𝐚𝐭𝐭𝐞𝐫?

We run 𝐚𝐩𝐩𝐥𝐞𝐬-𝐭𝐨-𝐚𝐩𝐩𝐥𝐞𝐬 ablations spanning Ouro to Huginn. Huginn performs better overall, with the largest gains coming from the loop-in-the-middle (sandwich) design and input injection, though they provide different benefits.

Trained on 𝟓𝟎𝟎𝐁 tokens, an 𝟖𝐁-𝐀𝟎.𝟖𝐁 Huginn MoE approaches or surpasses a 𝟑𝟐𝐁-𝐀𝟑.𝟐𝐁 feedforward MoE on several reasoning benchmarks, including GSM8K (83.6% vs. 80.8%), while using 𝟕𝟓% 𝐟𝐞𝐰𝐞𝐫 resident parameters under 𝐦𝐚𝐭𝐜𝐡𝐞𝐝 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 FLOPs.

More details and the blog link in the thread ↓

[Benhao Huang @huskydogewoof]

Great work! Here are some useful conclusions about loop architecture in their tech report:

1. Train loop models from scratch is better. Upcycling a pretrained standard Transformer into a looped architecture performs significantly worse.

2. Two passes appear to be the sweet spot. Additional passes bring only marginal gains, while substantially slowing training and making optimization less stable. (echoes loopie by @gaozitian123 from Ubiquant)

3. KV sharing across loop passes is not free. It cuts KV-cache memory by half, but consistently underperforms the full loop with separate KV states.

4. Loop depth cannot be tuned in isolation. Stack depth and hidden width need to be co-designed with the number of passes to balance capability, training cost, and inference efficiency.

interesting enough, though the underlying ablation tables are unfortunately not shown (

[Francesco Sacco @FrancescoSacco1]

New high quality blog post: "How to train a Frontier-Level world model" in collaboration with @reactorworld

This is going to be very useful for practitioners

[Zhengyang Geng @ZhengyangGeng]

If compression is intelligence, text cannot be the final for superintelligence. Text is already a remarkable human-made compression of thought and experience; but it is still an interface shaped for human communication, not necessarily the right from for intelligence beyond us.

With Latent Thought Flows, we try to compress text further into a continuous latent space that could support generative modeling. We actually bet on representation learning a lot by compression.

Will the Bitter Lesson win agian?

It has been absolute fun working on this with @mihirp98!

[Zhengyang Geng @ZhengyangGeng]

Thanks Sander for sharing our fun exploratory project and for the thoughtful comments!

I agree: continuous vs. discrete gives us a common modeling interface, but it is not the whole story. At the core is how each modality should be characterized, and at what level of abstraction. Ultimately, we should let the data tell us.

Our bet is that representations for text, code, and math can become more compact and generatively modelable. Whether that narrows the gap with audiovisual representations—or reveals that audiovisual latents are simply too low-level—is indeed an interesting question.

We’re absolutely going to keep playing with this. For me, the keyword is simply: representation!

[Jacob Effron @jacobeffron]

I sat down with @lukaszkaiser to get into whether the architecture he helped invent is actually enough, and what's next in generalization, coding agents, RL and more. Lukasz co-authored "Attention Is All You Need," the paper that introduced the transformer and worked on reasoning models at OpenAI so he’s been a key part of major shifts in the field. We hit on:

▪️ The case for and against a new architecture coming after the transformer
▪️ What’s required for model generalization in the physical world
▪️ How much coding agents have improved his AI research productivity
▪️ The next domains for RL
▪️ Why Anthropic initially won coding
▪️ Future research directions he’s excited about

0:00 Intro
1:12 Transformers vs. Human Learning
8:37 How Do We Get Physical World Generalization?
10:52 What Comes After Transformers
13:59 How Much Have Agents Improved Lukasz's AI Research Productivity?
17:21 How Close Is an AI Research Intern?
26:06 RL Beyond Verifiable Tasks
35:38 App Companies: Build Models or Lean on Labs?
46:21 Multimodal Is Still Missing Something
49:46 OpenAI's Bet on Reasoning
55:26 The AI Coding Wars
59:26 Focus vs. Keeping Embers Burning
1:02:09 Open Source vs. Closed Source Gap
1:05:15 Quickfire

YouTube: https://t.co/wetErBD4B8
Spotify: https://t.co/FSYWR39ep2
Apple: https://t.co/j2omVI8xEa

[will depue @willdepue]

A Stargate for Data

Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data.

At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger clusters and datasets, and received incredible new capabilities in return.

But this exponential scaling is bound to hit some limits. Oddly enough, compute has compounded fairly smoothly without limit, with trillions flowing into hypercluster buildout. Instead, we’re starting to hit the limits of an exponential demand for data. Gone are the days of being purely in the compute-limited regime, where we had effectively infinite internet data but never enough GPUs, we’re now entering a data-limited regime.

Luckily, this limitation is coinciding with staggering improvements in AI capabilities. Incredibly, we seem to have a real line of sight towards automating a majority of knowledge work with the methods we have today. RL + pretraining, and the data for each, will be generally sufficient to achieve most economically valuable tasks, given some minimal algorithmic progress and continued compute scaling.

In a data-limited world, economic progress & scientific acceleration will be directly bottlenecked by our coverage in each domain. We need to see data collection as imperative, deserving the same civilizational ambition we’ve given compute.

The internet as a one-time subsidy

It’s underrated how much all progress in AI owes everything to the blessing of the internet, this one-time civilizational subsidy to deep learning, decades of unintentional accumulation of a perfect dataset: every book, blog post, image, video, paper, discussion, etc. all digitized and freely available. Without the internet, we’d likely see comparably minimal progress in AI today, and in fact, if you notice where systems currently underperform, it’s almost always a domain where web coverage is limited and data is private, expensive, non-digitized, or non-existent.

But we’re running out of it. There are only about 300 trillion tokens of useful public human text, and the internet doesn’t produce nearly enough new high-quality data to match what scaling demands — we’re soon to hit the limits of public data for pretraining. And though the advent of RL bought us reprieve — chain-of-thought RL needed a new form of untapped data, gradable math & coding tasks, also available online — we’re quickly running dry of hard tasks for RL as well.

Why do we need so much data anyways? Humans learn comparably in far less time, needing just one textbook where language models might need the equivalent of hundreds to learn a new topic. It’s possible we discover methods that are massively more data efficient — synthetic data, data efficient architectures, other exotic algorithms — but fundamental progress is slow and highly unpredictable, and the recipe we have just works today.

And, while I’m wary of getting too deep here, even arbitrary data efficiency can’t replace data that just doesn’t exist in the first place. There’s a massive amount of missing information on the web: the dark matter of the internet — tacit knowledge, undocumented processes, etc. — most of which was never published and lives only inside organizations, the physical world, or just in people’s heads. I’ll leave it here and say, for reasons far longer than I can fit in this post [1], it’s best to operate on the assumption that our insatiable desire for data will continue as it has for the last decade.

There will be >$100B/year in data spend by 2030

We’re not screwed yet, of course. Only a fraction of useful data in the world is on the public internet, the rest is stored inside private datasets, corporations, personal archives, universities, governments, and otherwise. Labs can and will continue to license these private datasets, or create them from scratch, like Anthropic’s book scanning project. And we’ll increasingly task human experts to manufacture new high-quality data, with a large fraction of hard RL training tasks already being sourced this way.

But collecting this data, unlike before, will be expensive. As the free internet dries up and demand for data rises, we should see labs investing equally in data as compute, likely spending a significant fraction of their compute budgets on data. As we see trillions spent on compute, we should also expect hundreds of billions spent on data (human data & collection budgets), given their equivalent importance. And, notably, data spend is already tracking this way: total data spend across vendors, not counting internal lab efforts, is already roughly $7 billion per year. It’s quite reasonable we’ll see >10x by 2030.

Data is the moat

Data becoming increasingly private will also majorly shift the competitive landscape. While compute is a commodity — everyone buys the same chips and builds the same clusters — data really isn��t. The big reason why frontier models have felt eerily similar to one another, until now, is they were trained on substantially the same internet (pretraining data variability across labs seems pretty low). As labs diverge onto more exclusive, manually collected corpora, I think models will begin to increasingly diverge.

OpenAI pulling ahead in mathematics and Anthropic in cybersecurity isn’t an accident. I really think laser-focused collection of high-quality midtraining tokens, custom RL tasks, environments, with dedicated research effort, has driven much of the visible progress in the last year. James Betker has an excellent blog about “the ‘it’ in a model is the dataset”: model architecture and compute buy you efficiency and order-of-magnitude performance, but ultimately, models, of any architecture, are such incredible approximators of their dataset that the core meat of a model boils down to just that, nothing else. Data is a major moat.

AGI long, ASI short

As I’ve tweeted before, I’m confident that, despite the narrative, the data labeling industry will continue to fuel great businesses and be an excellent AGI long, ASI short. The argument is just: By the time the AGI labs no longer need data, it’s probably over for everything else too [2]. In this frame, the last companies left should be the data companies, as the last speck of economically relevant data is sucked in. And these companies are already among some of the fastest-growing companies in history: Mercor, founded three years ago, is rumored to be doing $2 billion in revenue with something like a few million expert labelers under contract.

While these businesses are very non-stationary, what type of data is needed shifts constantly, I don’t think that diminishes their value. The long-tail of the economy is long, and the value isn’t diminishing as you extend farther into more obscure information: as models get more capable, the value of the marginal dataset goes up, not down. Automating a full job means covering its full distribution of tasks, tools, edge-cases, and long-horizon loops. There’s some O-ring logic to it: a dataset that buys a 1% bump can justify a previously unjustifiable collection cost when it’s the difference between a system that does 99% of a job and one that does all of it [3].

The competitive dynamics of the data industry are still evolving but as demand for data is increasingly niche, ultra high-quality, expert-generated, I think we’ll see real consolidation. Again, contra-narrative, we’ll probably see true competitive differentiation built on brand, quality control of data (which, from personal experience, can vary massively), as well as in network effects from the talent networks themselves over time. We’ve already seen rapidly shifting data type demand work in favor of incumbents, benefiting those with early knowledge of where the market is headed.

The binding constraint

It’s truly remarkable that we seem to have the recipe — pretraining + RL — to absorb most economically valuable work, despite being far from a lot of what we expected from “AGI”. The same way chess engines revealed we never needed general intelligence to solve chess, as we originally thought, we’ll soon realize that software, mathematics, and the vast majority of the economy (including physical, just running ~3 years behind!) are the same. If recursive self-improvement or some other algorithmic breakthrough arrives, that’s wonderful, but we really don’t have to wait for it. The binding constraint between here and an automated economy isn’t that, it’s data coverage: every app, workflow, edge case, process, etc. sitting in private stores or someone’s head.

Ultimately, while we make tremendous strides in more efficient model architectures, and clusters like Stargate equip us with zettaflop-scale compute, we really aren’t making rapid progress collecting the data we lack.

We’ll soon live in a world where we have the methods & compute to accelerate scientific progress or economic growth, but not the data. And we’re already there today: frontier models would surely be as good at accounting/many medical tasks/legal advice as they are at software engineering if we only had the same pretraining & RL coverage as we did for code.

I really want to drill this in: The speed at which we automate the economy is going to be directly rate-limited by our ability to collect data about it.

Worth noting that under this assumption, with data as defensible and directly proportional to economic & scientific progress, data should also be considered a national strategic asset like compute. Imagine what we’d do in a world where we had a Manhattan Project-effort for AI and needed to mobilize data collection as a limiting factor. We should be concerned about China, with greater state capacity and authoritarian economic control, being capable of mobilizing data collection at national scale, potentially compounding their economy and scientific output faster than us down the line.

A Stargate for data

I’m leaving my complete ideas for a future post, as this one is already far too long, so I’d really like to pose the question here. Stargate exists because we organized trillions of dollars, international strategy, gigawatts around compute as a fundamental ingredient. What would equivalent ambition look like for data?

Obviously, scaling data collection, a heterogeneous mass of information across the economy, isn’t going to be as clear as scaling compute, as a homogenous infrastructural effort. A core division will be first, coverage — all uncaptured knowledge sitting across the economy/science/physical world and all that simply isn’t recorded — and, secondly, sheer volume in the domains we already train on: more hard math tasks, more high-quality web text, way more coding data, more legal drafts, etc.

I have a post coming soon which breaks down my proposals. There’s a lot of room for creativity. Quickly, we’ll probably want to start with a deep census of what we have and what we’re missing, predict what the 2030 model will still be bad at and work backward to what we should be collecting today. You can probably license a large amount, leveraging high lab valuations to buy datasets or companies altogether. There’s an adversarial nature to a lot of this collection with firms, so there’s lots of engineering to do this correctly. We should go convince important companies to turn off deletion policies, even if we’re not buying from them yet. Data flywheels in consumer products will be massive. Confidential training, government legislation for grant-funded research, running companies at a loss for their data, etc.

We’re headed towards hundreds of billions in expenditure, national prioritization, and major data limitation on the horizon. We have a great opportunity to think creatively about what a megaproject for data would look like: How do we, deliberately this time, construct the next internet’s worth of data?

Footnotes:

[1]: I’ll probably soon publish my much longer post explaining my position on data efficiency and why the value of this data is still pretty high in most worlds regardless of new algorithms.

[2]: The “AGI freeroll” bet: heads you win, tails ASI flips the world upside down anyways.

[3]: We already see a glint of validation of this point, given the data market is strongly tilting towards ultra-high-quality agentic data, rather than unskilled labeling — niche expert workflows, live environments, and evaluations requiring increasingly obscure talent & knowledge — yet shows increasing, not decreasing, revenues.

[Sander Dieleman @sedielem]

Here's a cool piece of LLM lore: the original scaling laws were wrong due to a bug, which probably led to a lot of wasted compute on oversized undertrained models 🫣 (and that was before we even started properly accounting for inference cost!)

[Raj Dabre @prajdabre]

Kinda reminds me of another work on language modeling with pixels. They go tokenizer free on the encoder this way. Neat idea tbh.

Back then I wondered how we would get pure pixel to pixel but generative text rendering in images was not well addressed back then.

Link in comments.