1st pass: My sleepy brain couldn't understand anything..
2nd pass: Got a decent hold on the equations..
Maybe by third pass, I'd be able to internalize a thing or 2 from this episode..
This is great format @dwarkesh_sp! And 🫡🫡 to @reinerpope!
Did a very different format with @reinerpope – a blackboard lecture where he walks through how frontier LLMs are trained and served.
It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk.
It’s a bit technical, but I encourage you to hang in there - it’s really worth it.
There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him.
Recommend watching this one on YouTube so you can see the chalkboard.
0:00:00 – How batch size affects token cost and speed
0:31:59 – How MoE models are laid out across GPU racks
0:47:02 – How pipeline parallelism spreads model layers across racks
1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.”
1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal
1:32:52 – Deducing long context memory costs from API pricing
2:03:52 – Convergent evolution between neural nets and cryptography
If agents can’t solve a bug, your observability is broken.
Close logging gaps. Give agents access to logs, databases, and the codebase. Create the right skill files.
Humans should fix the setup, not the bug
@MetacriticCap With bedrock, you'd miss out on Claude's desktop app, Cowork, Claude Chrome Extension, Claude Code web, slack app, claude code on mobile, etc etc.. Although, some of these might be on the way to bedrock as well..
After seventeen incredible years, my latest chapter at Tesla has come to a close. Words won’t do justice to how fulfilling the experience has been, but I’ll try anyway:
Tesla barely survived Christmas 2008. I started a few days later in our Finance team, under an ongoing “Tesla Deathwatch”. I slept under my desk in San Carlos, CA at least once, and I wasn’t the only one. There are many companies with hard-working and talented employees, but few have the level of commitment and collaboration of the Tesla team. In retrospect, this should have been an obvious predictor of the successes that would follow. This is as true today as it was in 2009.
To my former Finance team: You are heroes within a company full of heroics, given the full-body workout that is your daily job. Take an Accounting or Finance textbook off the shelf and flip to a random page - Tesla is undoubtedly manifesting the underlying concept in the real world. From selling hardware and software under various business models, managing assets of all types (including digital ones), to pursuing continuous investment into an ambitious future throughout the world, you make the impossible look easy.
To my colleagues across the rest of the company - I am grateful for the time that you have spent to educate me, including during my stints outside of the Finance team. You have started with one of Tesla’s strongest advantages - the strongest talent across engineering, manufacturing/operations, and customer-facing functions - and have turned it into an unstoppable force via your “one team” attitude. For me, late-night sessions on topics ranging from the physics of a brake rotor to the training of a neural network proved more valuable than any classroom experience that came before, and made the company stronger as a result. Going to work is not supposed to be this much fun.
@elonmusk : A heartfelt thanks for your endless love of humanity, and for demonstrating the power of thinking from first principles at all times, about all things. When Abundance is achieved and money ceases to have meaning, these lessons will be the most valuable commodity in our economy.
To the outside world, who may not have experienced the above first-hand: Remember that Tesla’s mission is so ambitious and complex that any narrative about the company is naturally an oversimplification. Seek the truth about the company at all times. And support it in any way that you can! There are few higher callings/better uses of your time.
If I ever trip slightly while walking, I make sure to look back and down at the ground so that the people around me know that I'm normally great at walking, but in this particular instance there was something wrong with the ground.
I miss San Francisco.
I miss Crissy Field. The Marina Green.
I miss that walk along the water at sunset, with all that green to your left, the water to your right, and the sight of that huge surreal orange bridge up ahead.
The Italian sandwich spots in north beach. The insane views from Russian Hill.
Getting lost in the Presidio, or Golden Gate Park for hours at a time.
The parrots flying above Alta Plaza Park.
I miss the sand dunes at Ocean Beach.
All those little stores in the mission that sell the most random things that you can’t help but check out over and over again, even though you’re not really sure what they are.
I miss running into some of the best coffee spots in the world everywhere you turn.
The feeling that you’ve gone back in time as you wander through the Haight-Ashbury and walk by stores that somehow survive selling nothing but tie-dye clothes.
I miss hiking around Lands End, in disbelief that place exists on earth.
I miss the time-worn but perfect hole-in-the-wall Chinese food restaurant you can’t get enough of.
I miss walking into the Ferry Building on a Saturday morning and wanting to sample from every single one of those little shops, and then existing out to a picture perfect farmers market along the water.
I miss that stunning downtown view that rises you out of nowhere as you head up highway 280.
I miss the electric energy of a crowded Dolores Park on a sunny day, and guessing just what crazy thing that next vendor is going to walk by with.
I miss the occasional movie at the Castro Theater, and that hilarious vibe as you’re heading inside.
I miss walking up to one of the most beautiful ballparks ever built, and that smell of garlic that sharply greets you as you enter.
I miss the throwback steakhouse vibe of The House of Prime Rib.
Or occasionally putting on my tourist hat and heading down to Fisherman’s Wharf or over to Alcatraz; two places where you’re guaranteed to never run into anybody you know.
I miss zipping down Franklin St and timing it so the lights all turn green just as your car approaches the intersection.
I miss the authenticity of Chinatown.
I miss how the wind sounds as you ride the ferry across the bay to Sausalito. Talk about a bucket list experience.
I even miss hearing those loud kids that crowd The Tipsy Pig all day, screaming at the top of their lungs as that fifth drink starts to hit them.
I miss that it never gets too hot, and that picturesque layer of fog that settles just above the water, even when it hangs around for just a few weeks too long.
I miss the eclectic mix that makes up the people of San Francisco, and the passion they have for their city.
And most of all, I miss the enthusiasm and the optimism, even within a city that has always had its challenges.
The people who live there are there because they know there are better days ahead.
They know they live in a special place.
It’s been down before, but there are just too many great things about it to ever count it out.
San Francisco will be back.
And it will be better than ever.
It’s an American national treasure.
It’s a place we should all be rooting for.
Announcing a new Claude Code feature: Remote Control. It's rolling out now to Max users in research preview. Try it with /remote-control
Start local sessions from the terminal, then continue them from your phone. Take a walk, see the sun, walk your dog without losing your flow.