Scott Condron

Helping build AI dev tools at @weights_biases. I post about AI, data visualisation and the stuff I’m working on at wandb.

[Scott Condron @_ScottCondron]

Here's an animation of a @PyTorch DataLoader.

It turns your dataset into a shuffled, batched tensors iterator.

(This is my first animation using @manim_community, the community fork of @3blue1brown's manim)

Here's a little summary of the different parts for those curious:
1/5

[Deok @deok_filho]

nice try

we spun up 10 THOUSAND sandboxes, @opencode running in each one pointing towards our serverless inference endpoint to get ASCII art

I'm feeling like an artist today 🪏

CoreWeave Scale

[Scott Condron @_ScottCondron]

@code_star What if it’s pronounced like Jorge, with a throat clearing for-ge [x]?

[Canva @canva]

Introducing Canva Craft 🚀

Extraordinary image generation is now available to everyone, powered by our first frontier model built specifically for design.

With Craft, Canva is now the world's second most advanced AI lab for image generation on @DesignArena's global leaderboard.

See what it can do 👇

[Scott Condron @_ScottCondron]

A heatmap view of agent swarms activity on a bots-only runescape server. They trade and compete for resources (and scam each other).
Great talk by @maxbittker
https://t.co/woOwZ2ryZM

[max @maxbittker]

Runescape Bench talk now up on youtube!
Covers results from the benchmark, design of the harness, and economic swarm experiments I've been doing.
https://t.co/gkgXdAI3eZ
Thanks @recursecenter

[Scott Condron @_ScottCondron]

> In our public run, each ~80 s training step hides behind ~40 min of agentic rollouts, and the only pause is a ~4 s weight broadcast to 96 SGLang engines.

@IFM_AI shared a public wandb run which show where the infra bottlenecks for this kind of large scale RL training. While you're doing the backprop, you can be doing lots of rollouts with the last steps weights.

[Institute of Foundation Models @IFM_AI]

We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.

Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.

Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.

RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.

Code: https://t.co/eA46QBpfTx
Run: https://t.co/cd4tOV3OXv

[Deok @deok_filho]

Danke @Aleph__Alpha !
Fantastic article that all so called "Sandbox providers" should be reading.

Sandboxes are not just for running cloud agents. We see massive sandbox volume in research. Thousands of concurrent sandboxes per run. Many runs per researcher. Many researchers per company.

I talked about the balancing act of keeping throughput high for the Trainer in a webinar this Tuesday. Watch it in thread 🧵

The SPICE must flow
https://t.co/kTQNrgipXm

[Scott Condron @_ScottCondron]

@sh_reya Congrat!!

[Scott Condron @_ScottCondron]

@morgymcg Oh ok, so it’s a new version of senpai with a different style of auto-research communication? I was thinking it was a library or paper or something

[Scott Condron @_ScottCondron]

Exciting result from Mirendil. They built a pretrained base model with $10mil in compute and 1 engineer within 3.5 months, and an agent wrote 99% of the code and ran the experiments - now they'll use that model to improve future models
👏

[Behnam Neyshabur @bneyshabur]

We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:

Pretraining at scale without a pretraining team.

Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.

Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?

Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.

The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.

We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.

We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!

[Scott Condron @_ScottCondron]

@altryne Always-on agents?

[Scott Condron @_ScottCondron]

Nice mini repo in here for auto-research on your own machine. Auto-research can get spenny quickly, not anymore :)

[Cong Lu @cong_ml]

1/ From OpenAI's Navier–Stokes work to AlphaEvolve's faster algorithms, AI research is getting very real.

Now, with Recursive Flint, we can experience that magic locally on our own laptops 💻

The whole research loop runs locally with a selection of tasks! Code + demos 🧵

[Scott Condron @_ScottCondron]

Love seeing the open problems in continuous learning laid out this clearly. Line of sight to “owning your intelligence” and reasons to move away from relying on models from the big labs. Not all problems need model weight changes, but lots of important ones do.

> there are major open challenges around knowledge acquisition, harness-weight tradeoffs, user modeling, dynamic underspecified preferences, updating and planning across timescales, selective forgetting, data efficiency, learning from golden answers, and company-scale simulation. this is the roadmap, roughly speaking.

[will brown @willcb]

applied research at prime intellect is both a client-facing team and a fundamental research team. continual learning is our north star. we view this dual approach as necessary in order to study the core challenges faced by models which self-improve in production. the real world is big and messy.

we spend part of our our time working with customers to train SOTA models for specialized domains, and we automate and productize our work as we go. we also do open-ended research on novel methods for self-improvement and long-horizon memory, and we get to experiment with data that no other lab gets to see.

today, the most readily-deployable form of continual learning is converting "exhaust" (agent traces, logs, feedback) into robust environments and tasks for continual RL. but this is not the final form. there are major open challenges around knowledge acquisition, harness-weight tradeoffs, user modeling, dynamic underspecified preferences, updating and planning across timescales, selective forgetting, data efficiency, learning from golden answers, and company-scale simulation. this is the roadmap, roughly speaking.

some of these may require changes to architecture, or parameter-efficient methods, or fundamentally new algorithms. we have our hypotheses and early evidence, but everything is on the table. the world is our oyster.

we are betting on a radically different future of what ai deployment will look like, and we can see the path more clearly every day. it is a world where agents learn like humans do, with billions of individual models compounding their learnings from the real world continuously.

we're hiring incredibly ambitious researchers in SF, NY, and London. come join us.

[Scott Condron @_ScottCondron]

Continual learning and auto-research is imminent it seems 🤯

[Robert Lange @RobertTLange]

Two landmark releases today on recursive self-improvement! What a timeline 🚀

Mirendil (@mirendil) reports pretraining Mirendil-Base, a 100B–500B coding model, from scratch in 3.5 months with one researcher and under $10M of compute; small-scale experiments included. Agents wrote over 99% of the code and ran the experiments, while the researcher chose directions and decided which results to trust. Larger versions are already training. Impressive leverage for a single person.

https://t.co/W7tarKwYU0

Recursive (@Recursive_SI) released Flint, an open-weights 40B coding model for fast local use. They built it from scratch in under 4 months with their automated research system, the Eureka Machine. Great to see autoresearch ship an actual model people can run.

https://t.co/8fwuh15Uoy

The framing seems to differ a little. Mirendil is reducing the people, compute and time needed per model, with a human directing agents. Recursive's Eureka Machine aims to automate more of the research loop itself. Both start with coding models, on the view that better coding speeds up AI research.

[CoreWeave @CoreWeave]

Bringing production-scale AI inference to life takes a strong open ecosystem. As a founding member of the project to now releasing llm-d 1.0, we're thrilled to support this community and fuel the future of open innovation 💪

[Scott Condron @_ScottCondron]

Auto-research on idle GPU capacity

Every idle GPU is spice that isn't flowing.
And the spice must flow.

One idea to keep your GPUs busy is an auto-research pipeline: a queue of different ideas for agents to try whenever the GPUs are not doing higher priority work.

On @CoreWeave you can build this from:

- SUNK (Slurm on Kubernetes) to share nodes between priority jobs and agents, with preemption so priority work gets done
- GPU sandboxes to run agent-written code in isolation
- Spot capacity for cheap extra capacity 
- Dedicated inference plus Dynamo agent_hints to rank the agents' own model calls below production traffic

And you can see what the agents are doing: traces, evals and training runs in one place.

[Scott Condron @_ScottCondron]

@trq212 No they just need a new control plane

[Scott Condron @_ScottCondron]

I vibecoded this UI to manage my agents

[Scott Condron @_ScottCondron]

@just_cameron Each of these tracks is actually a control plane

[Scott Condron @_ScottCondron]

I vibecoded this UI to manage my agents

[Scott Condron @_ScottCondron]

@jeroaranda Kanbahn is genius. It’s missing the photorealistic train simulation though

[Scott Condron @_ScottCondron]

I vibecoded this UI to manage my agents