Personal update: I’ve decided to leave OpenAI.
I’m proud to have been part of the custom chip program and grateful to everyone I got to build with and learn from along the way. The density of hardware talent on that team is extraordinary, and I don't think there's a better chip design team anywhere. It's been a wild journey from second hardware hire, 2.4 years ago, to now, and I'm excited to watch these chips become one of the most important engines of AGI.
At the same time, I haven’t been able to shake the pull to climb a new mountain from the bottom again!
I joined @AnthropicAI this week because I was deeply impressed with the team’s talent, values, and ambition, and I'm already energized by the pace and intensity of the past few days. It’s time to build.
Congrats on the tentative proof of GRH @__alpoge__ !!
I wish I had the mathematical background to appreciate the beauty up close, but at least I will be in awe from afar.
Working with @AMD, our models identified up to 14% PPA improvement on flagship GPU IP in less than five months—building on designs AMD’s engineers had already spent years optimizing.
The same approach helped improve Jalapeño, our first custom inference chip, with gains verified on real silicon.
New evaluations also show our models getting better at certain chip design tasks.
We’re sharing these results and how we’re working with @Synopsys to bring our model capabilities and their trusted EDA design and validation tools to more chip designers.
https://t.co/4BNcPIqfIQ
just finished omakase dinner party 2! thanks for coming @caithrin@arankomatsuzaki@gordic_aleksa@saragordic@raghavrmehta@ethanelasky@edgeofarete@buzzcai84@itsclivetime plus Alex and Melissa! food finished way too quickly for photos
dinner menu:
- Korean-Spanish style paella with kimchi, cashews and apple served in inari
- Sweet pepper jam, brie, peaches and prosciutto on crackers
- Dijon mustard, mushroom, caramelised onion butter tomato tagliatelle
- Kasama chocolate from Caithrin
- Biscoff granola crumble with Ben & Jerry ice cream
- Frozen strawberries and bananas from Sara
- Fresh raspberries from Ethan
- Indomie hahaha
- mocktail of the night: sparkling jasmine, lime and lemon water with rose and raspberries
I started writing a blog. Not sure if I'll manage to keep up.
First few articles are about how I think we should be doing HW-SW "co-design" with an example of exp2 HW - https://t.co/G2pLkzuIZ3
Kimi intentionally pacing the open weights cyber frontier is a big win. Cyber is defense-dominant long term (i.e. every bug gets fixed), but it’s looking like a nasty transition in the short term, so gradual diffusion is 100% the way.
Moonshot is intentionally throttling cyber capabilities. From the blog;
'First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes.'
What happens when the best writers, narrative shapers and world builders in SF gather for dinner? We’re about to find out.
Formally kicking off my dinner party series for those in the know and for the best builders shaping the stories, ideas, and worlds of tomorrow.
IYKYK :)
PS. Giving due credit to @majamediaco
for coining the term world builder!
@zarazhangrui I’ve been adding elements of Singapore’s cultural heritage and San Francisco to mine! Plus a few hidden Easter eggs that are personally meaningful
Link: https://t.co/N3DVelSDPN
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya@noahdgoodman, and @YoavLevine.