Benjamin F Spector

pilot of @flappyairplanes. i make ml go brr.

[Benjamin F Spector @spectorb]

@plugyawn V cool. Dm me, would be glad to help!

[Benjamin F Spector @spectorb]

@Enigma_AI Lovely writeup!

[Benjamin F Spector @spectorb]

@Enigma_AI Congratulations! This is awesome

[Benjamin F Spector @spectorb]

@tinahhong Congrats!

[Benjamin F Spector @spectorb]

@evayzh you watch even this at 2x?

[Benjamin F Spector @spectorb]

math checks out!

hadn't thought about it this way before your post but I think another way to view this is that you really just want to max the total shoreline of the system (provided enough low-latency networking). In your example, I think what the smaller-die approach is getting you is really shoreline/$, and allocating enough shoreline to networking allows the collective shoreline to behave as a coherent system.

[Benjamin F Spector @spectorb]

@levidiamode @karpathy Thanks very much @levidiamode , this is so incredibly kind! Happy to do a follow up video with you if you'd like -- would be glad to talk a bit about Blackwell, megakernels, etc, if useful.

[Benjamin F Spector @spectorb]

i propose, tonight only, we get to light off a nuke over the pacific. for america's 250th the world can let us have one special firework, just for the occasion

[Benjamin F Spector @spectorb]

- I saw the whole throughout-interactivity pareto frontier be characterized in real time on the A0 silicon, with the token outputs streamed too.
- I went through their codebase for how they do megakernels and thought it was, genuinely, very well done.

I actually agree it would be good if they put more of their numbers out in public. (FWIW I did encourage them to. I know why they haven't yet and it does make sense but I am not sure it's correct on net.) But I still dislike your insinuation that everyone around the table is a paid shill, which I think is insulting and also just wrong.

[Benjamin F Spector @spectorb]

@__tinygrad__ @Etched This is stupid and you should feel bad. I have seen the actual hardware generating tokens at astonishing speeds. You must think very little of the people on this list to believe they would squander their reputations for a few dollars.

[Benjamin F Spector @spectorb]

Had a chance to dive deep into this with the Etched team, and came away extremely impressed. It's a good chip sir.

Their programming model around megakernels is also very simple and controllable and quite well done. Focus here on flop density + scale up seems right. Big kudos to the Etched team!

[Benjamin F Spector @spectorb]

@InferizeAI Awesome stuff, can't wait to play with it!

[Benjamin F Spector @spectorb]

@EngramLab Congrats! Very excited to play with what you guys are building!

[Benjamin F Spector @spectorb]

All reliable clusters are alike; each unreliable cluster is unreliable in its own way.

[Benjamin F Spector @spectorb]

@hamzaelshafie What a wonderful, in-depth post! I'm immensely appreciative of the effort that must have gone into this. I've added a link to the readme of the TK repo. And footnote (2) is a good idea -- an xor instead of down shuffle would probably save a few instructions!

[will depue @willdepue]

megakernels remain underrated. if you haven’t dug into them before go look them up! flappy seems to be hinting at some really powerful training megakernel stuff which is sick
ex: fully contained training megakernel could be great for automated research

[Benjamin F Spector @spectorb]

Fun to share a little taste of some of the things we've been working on with @sequoia

[Flapping Airplanes @flappyairplanes]

(1/5) Great to be at @sequoia to give a sneak peek of one of our research directions!

TL;DR one path to data-efficiency may be to “abuse GPUs like they’ve never been abused before”

[Hayden Prairie @hayden_prairie]

We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.

Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size.

Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem!

🧵👇

[Benjamin F Spector @spectorb]

@dylan522p @SemiAnalysis_ @Kurnalsalts Ah I meant they'd get physically larger to allow for lower power consumption. The whole thing has to run on like 100W or something? Including memory and whatnot. But this was an offhand comment, I am far from confident I am right.