Core Automation

Building systems to automate and optimize

[Core Automation @coreauto]

building up the tech tree

[Core Automation @coreauto]

We have little time here but may be doable in 126 days

[Jonathan Frankle @jefrankle]

Congrats to @srush_nlp for increasing the amount he's going to owe me when he loses this bet in 126 days 🤗🤗🤗

[Core Automation @coreauto]

We should take it seriously.

The main limitation is that there are actually very few neolabs putting serious compute into fundamental research instead of doing rlaas or open weight transformers, but there are few who are doing it, are determined and moving fast.

[Theo Jaffee @theojaffee]

How seriously should we take the possibility of one of the neolabs making some wild algorithmic breakthrough that puts it ahead of OpenAI and Anthropic?

[Core Automation @coreauto]

@infoxiao @clankrmedia This could be us

[Core Automation @coreauto]

How to live your life:

Meeting Mondays
Thinking Tuesdays
Work Wednesdays
Trying Hard THursdays
Automation Fridays
Sleeping Saurdays
CUDA Sundays

And repeat!

[Core Automation @coreauto]

A prompt that launched a thousand training runs

[Core Automation @coreauto]

Even in natural language you need to continually learn

[Joanne Jang @joannejang]

learned this from the youth:

if something sucks, you no longer say it's ass. you say it's "cheeks"

e.g. "this model is cheeks"

[Core Automation @coreauto]

AI discourse often focuses on scaling because of eye popping numbers spent on datacenter compute. Credit assignment is very difficult

Most people from outside big labs and even many inside get this credit assignment wrong.

It’s worth doing a simple thought experiment:

It’s may 2020. GPT-3 paper just got released.

We have two diverging timelines:
a) we have the same algorithmic progress that we did since gpt3, but we cannot spend more compute on training models than was spent on gpt3
b) we keep scaling up gpt3 and we spend as much compute as we did on gpt5.6, but on gpt3 training system

Reality is that model from timeline a) beats model from timeline b) on every axis and it’s not even close

[Core Automation @coreauto]

Power of friendship is one of few durable moats

[rohan anil @_arohan_]

Power of friendship was always going to be one of the new pretraining axis

[Core Automation @coreauto]

We are very proud to co-organize open participation competition inviting scientists from around the world on one of the deep learning problems that is still open: https://t.co/118fiCHEkc

[Core Automation @coreauto]

In deep learning, deeper is almost always better.

If it isn’t, that means you’re just not very good at optimization.

[Ben @SolidlySheafy]

One Layer Deeper is officially live!

The motivating idea is that some models just don’t want to learn. Most optimizer benchmarks ask how quickly you can train a given model, but baked into that question is the assumption that it can be trained (well) at all.

We think this matters especially for adaptive computation. Ideally, a model should be able to spend (far) more compute on harder problems and use more computation at test time than it spent on any given training instance. There are many possible ways to do this, and we don’t want to assume the answer is one specific recurrent or looped architecture.

Making this all work is really an architecture-optimizer co-design problem. Rather than fixing the optimizer, loss, and training setup, One Layer Deeper asks people to design all of these together.

For the competition task, we use repeated modular squaring, x^(2^T) mod N. The task is inherently serial because each squaring needs the residue produced by the previous one. When N is a semiprime, the only known way to skip ahead requires knowing its factorization. Increasing T therefore adds another genuinely dependent step without making the input or output longer.

We are very excited to work on this with @marksaroufim @_arohan_ and everyone on the @coreauto team.

Submissions: https://t.co/cJYJR1Djal
Blog Post: https://t.co/RnZWL35SMu

[Core Automation @coreauto]

CoreWarming party!

[Mark Saroufim @marksaroufim]

We had the @coreauto housewarming party a couple of weeks back. It was tons of fun and should hopefully give you a better idea of where we're spending some of our time improving model architectures and systems

00:00: Introduction
00:40: @MillionInt (@coreauto), Building the world’s most automated AI lab
12:10: James Demmel (@UCBerkeley), Dense Numerical Linear Algebra: A brief history and current trends
42:11: Christoph Klein (@nvidia), Dense Linear Algebra at NVIDIA
47:54: @marksaroufim (@coreauto), Linear Algebra Kernels for the Age of Research
51:43: @SolidlySheafy (@tilderesearch), One Layer Deeper…

[Core Automation @coreauto]

Nice!

[Shobeir Fakhraei @shobeirf]

An exciting talk on Long-Running Agents by @MillionInt @coreauto (previously @OpenAI ) #agenticaisummit

[Core Automation @coreauto]

Get up and try harder. Age of research is waiting for you

[Harmya @racerfunction]

watching my loss not even move an inch on my 999th attempt at one layer deeper competition by @tilderesearch x @coreauto

[Core Automation @coreauto]

Losing to hardware lottery is a skill issue

[Lucas Beyer (bl16) @giffmana]

I present to you the ultimate showdown.

In our left corner, we have the researchers complaining about the hardware lottery.

In our right corner, we have the infrastructure guys like below, with the exact opposite argument.

Let the battle be fair but harsh!

[Core Automation @coreauto]

Thank you!

[Shikhar @shikhargupta02]

Very insightful conversation with the founders of @coreauto - a pretraining maximalist and an RL maximalist.

Got a little glimpse of what they think the next step could be in terms of model architecture. A couple of points they made that I really liked -

- The models are being deployed in the real world but trained in synthetic envs. This gap needs to be closed. Hinting towards continual learning.

- Autoregressive decoding is inefficient and latent reasoning provides a good pathway to move away from that. Recurrent layers can provide better computational depth. Also makes sense why they are hosting the https://t.co/iyKIGuhoA5 competition.

- RL is very inefficient right now. We sample thousands of tokens to get 1 bit of info into the model.

- New architectures will need better optimizers as well to use all the info you have during training.

It's joyful listening to extremely technical founders speak their mind.

[Core Automation @coreauto]

So what’s coming after transformer?

[Sonya Huang 🐥 @sonyatweetybird]

Two of the people most responsible for scaling the transformer are now betting on a next act.

@MillionInt ran the Reasoning 🍓 team at OpenAI. @_arohan_ was a pre-training lead on Gemini after years at Google Brain and Anthropic. They just started @coreauto to find what comes next.

Their core argument: (1) models are trained in the lab but deployed in the real world and can't keep learning once they leave; (2) AI research is done by humans today but models will be able to explore and uncover new advances more rapidly and systematically (controversial but timely w this week's petition).

The conversation covers:

— why Jerry expected AGI in 2025 and what changed his mind
— the two kinds of learning from experience, and why RL only captures one
— the computational depth problem baked into today's architectures
— why the biggest labs can't afford to look for a transformer replacement
— the kernel competition where humans + $100K of coding agents found a 60x speedup no frontier model comes close to
— a definition of AGI you can actually test: a model that improves itself with no human in the loop

00:00 Introduction
01:46 Appreciating Transformers
02:44 Scaling Hits Limits
04:54 Why Architecture Matters
05:32 RL Reality Check
07:32 Test Time Learning
09:52 Economics Of Scaling
12:47 Why Start A Company
14:24 Rohan On Transformers
19:11 Computational Depth Problem
20:32 When Transformers Top Out
23:22 Beyond Reinforcement Learning
26:41 Optimization And Efficiency
34:24 Building An Automated Lab
39:45 Kernel Automation Roadmap

[Core Automation @coreauto]

At least one day a week should be a CUDA day.

Refine your skills and study the hardware.

Hardware will reward you in exchange with a harmony of flops.

[Core Automation @coreauto]

Exquisite

[rohan anil @_arohan_]

A new meme format dropped.

What do you think @coreauto