Sung Jae Bae

@spc

[David Hall @dlwh]

I've been thinking about where the data for the next generation of open models will come from.

The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. https://t.co/tSajZlkEh9

[nathan chen @nathancgy4]

For people who aren’t sure whether these research innovations actually matter, here’s a simple explanation of why labs like DeepSeek and Kimi keep producing them:

Within a year, once LLMs can handle most everyday tasks, no one will choose an expensive model when a comparable one costs half as much. (and, separately, cannot access your data.)

The research we have seen so far has already reduced costs by more than 10x compared with models from a year ago, and these gains will only compound further.

Setting aside hardware differences, model architecture is the only factor that mathematically determines how much it costs to generate a million tokens. The rest depends on better hardware and infrastructure design, which are also continually being improved through new research.

And doing research actually doesn't require a crazy amount of compute. Coming up with an idea costs nothing, while the cost of validating ideas at <10b level scales is also negligible compared to frontier training runs. Only when a idea survives the initial smaller scale validation will it be brought onto the scaling ladder. Talent and taste matter more here.

The closed-source labs are likely doing something similar too. This is another inevitable path, alongside capability hill climbing.

[david lietjauw (Visiting SF 🌉 until Oct 20th) @davidfromkansas]

Can an LLM build &amp; run a flourishing city?

I ported SimCity2000 and built a gaming harness that gives @claudeai and @OpenAI models tools for memory, planning, and reflection so they can learn how to grow successful cities.

So far, Claude has built a city of 28,000 residents!

[ali @waterloo_intern]

i usually find the arguments for and against the best type of agent harnesses extremely not bitter lesson pilled-- it becomes increasingly obvious that the next frontier model with the most naive harness would beat today's model with the most sophisticated harness possible.

(perhaps the only exception being agent swarms, if one considers that to be a harness)

this shifted my opinion quite a bit... it seems to me that one can get relatively strong uplift in a relatively short period of time if they know what to target, which is what @thealexker set out to target in this book.

as an aside, watching @zakkeener42 work with the equivalent of agent swarms - before agent swarms were a thing - was a thing of beauty and privilege.

[loops @_smitop]

Anthropic released their 2021 research roadmap earlier this year (because of copyright litigation); I think some people might find it interesting: https://t.co/CpilF8XguE

[Konstantin Dunas @hallerite]

we are going to train swarms.

read my essay to understand why.

[𝖬𝖺𝗁𝖽𝗂 𝖢𝗁. (bluesky:@mahdi.ch) @mahdi_tcs_]

The what now?!
https://t.co/xHsLmQJkff

[Sung Jae Bae @sunjaebae]

Let the agents grind

[Chanakya Ekbote @thecekbote]

Happy to share that Interleaved Head Attention (IHA) has been accepted at #NeurIPS2026! 🎉

In standard multi-head attention, heads remain isolated during attention. As a result, combining multiple relationships often has to happen across successive layers. 🧠🔗

IHA lets heads communicate before attention by constructing learned mixtures of their queries, keys, and values, creating richer within-layer interactions while preserving the standard attention operation. ⚙️✨

Across long-context retrieval and mathematical reasoning, IHA consistently improves over standard attention. 🚀

Huge thanks to my co-lead author @dvsaisurya and our amazing coauthors @rach_it_, @rish2k1, @Devvrit_Khatri, @brandfonbrener, and brilliant mentors @pliang279, Inderjit Dhillon, and Manzil Zaheer!

🎥 Quick overview below
📄 Paper: https://t.co/KiXCIwaAsw

[Lucas Beyer (bl16) @giffmana]

Paper thread! I recently read this paper from some FAIR colleagues and NYU folks in depth, so making a thread while it's fresh in my head.

The main point is seeing how far on the SMALL side we can go in scaling laws without losing fit or predictability, and what it takes?

[Rishabh Agarwal @agarwl_]

I often see people mic drop with "Bitter Lesson" without maybe understanding its implications.

The best interpretation I like is from @hwchung27 "We need scalable methods that can better leverage computation."

For example, LLMs are amazing technology but their current training itself adds in a lot of inductive bias, e.g., data distribution being good enough to simply clone.

RL faces it's own version of bitter lesson: where does that reward come from? Well today, it's mostly human ingenuity.

[elie @eliebakouch]

wow insane, they are literally livestreaming the RL training run of Mimo V2.6 Pro (1T, 42B active) and Flash (309B, 15B active) with per batch data/harness composition and a ton of internal training metrics

https://t.co/mn7Fqaq1LO

[Riley Walz @rtwlz]

Some news: 5 months ago I joined @coreauto, a small research lab focused on new ways for models to learn. I’m also hiring four interns to do something my younger self would've loved.

It started with a pretty unusual project. We set up a handful of small online businesses, and put an agent in charge. We gave each business some initial funding and a seed idea, and off they went, trying to sell real things to real customers while we watched. It’s about as open-ended and unforgiving a task as you can probably give to an agent, with countless decisions to make and effects that take time to become clear.

Each business got its own bank account, a debit card to pay suppliers, a web browser, an email inbox, a domain name, cloud infrastructure and a schedule. Every few hours, the agent woke up and decided what to do. It might update a landing page, adjust an ad campaign, reply to a customer, and then go back to sleep.

The agents built actual storefronts and apps. They bought ads, attracted visitors, and even made some sales. The agents acted as operators that kept a business moving, but we noticed they never really stepped back to strategize about what might move the needle. If people weren’t buying, why? Was the landing page unappealing? Or maybe the offer itself was the problem?

One amusing example was the very first order a customer placed. Our supplier had run out of stock. But it’s a dropshipping business, and thankfully tens of other suppliers carried the same product. Instead of trying another supplier, the agent almost immediately sent the customer a polite apology and attempted to issue a refund. Gah! So close!

The agents could handle much of the work of getting a business up and running. What they struggled with was deciding what to change when it wasn’t working.

That’s what we want to understand. We also want to see what changes when people take the lead. What does a business operator actually do all day when agents can do so much? What new work does this allow them to do? Which decisions can’t be delegated?

—

So this winter we're hiring 4 interns as Associate General Managers. Associate Product Manager internships are programs that give early career generalists a product to own. We want to try something similar, but with entire businesses.

You’ll create and grow businesses, with agents handling the routine work so you can focus on figuring out what matters. Each AGM gets a budget and our agent infrastructure that’s designed for this. Which ideas to try is up to you. Put more money behind what finds customers and keep refining.

Where you step in tells us what agents can't do yet. You'll be in our SF office with me and people I’m so fortunate to work with, like @_arohan_, @joannejang, @MillionInt, @marksaroufim and @juliacvillagra

We don't care much where you went to school or what your resume looks like. We’re looking for enterprising people with a drive to create, who’ve made money on the internet before and already use AI extensively to enable their ideas. If that sounds like you: https://t.co/GfiGCVtMYU

[Jack Neo Zheng @jacknzheng]

We just broke 2 NanoGPT world records

On track 3 optimization, we set 2 new world records on both the Muon and MuonH optimizers by 90 (-3%) and 60 (-2%) steps.

We implement a simple variant of single EMA momentum - using multiple EMA buffers and anneal momentum during training

[Jerry Tworek @MillionInt]

Big labs are moving away very slowly from transformers because they’re making small gradient steps in each iteration and iteration takes them multiple months.

You can just front run it by… taking big steps 😀

[Kevin Tang @_KevinTang]

Super Smash Bros Melee in MR.

Where the players fight on your own table as a platform. Even hang off the table!

On the Meta Quest.

[elie @eliebakouch]

Deepseek V4.1 Flash 552B total, 8/16B active with a new arch trained on 45T tokens, there are different active parameters for input/output tokens with the encoder/decoder arch, engram, new sparse attention, new mHC, native vision

very high benchmarks (beating K3), insane efficiency, and as always amazing tech report

this is probably the most novel arch i've seen in a while, pretty insane

[Sung Jae Bae @sunjaebae]

@KJHMiao @frankieli_ Let’s go congrats!!

[Fei-Fei Li @drfeifei]

The world of R&D is forking into two paths: the token-abundant research, and the token-starved research. The future is in the former - evidentially, the progress by today's top AI industry teams and neolabs is breathtaking, where researchers’ human brilliance is super charged by AI’s assistance. Every research university president should be reading this report and reflecting on what the future of higher education research should be.
https://t.co/y74Ai3mIhk