Btw Vaibhav Sooryavanshi is a prime example of what an Indian kid can achieve at such a young age when their parents support, guide and protect them from the society created herd mentality that pushes everyone into the same JEE/NEET rat race.
Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.
we have to be absolutely relentless about ourselves. no despair. no giving up. even if you’re slow, even if it’s scary. even if everyone else disagrees. only you need to understand. you know best where to give yourself slack and where not to. & you can’t lie to yourself
msft, goog, meta, & amazon are on track to spend ~$700b on ai infrastructure in 2026.
this kinda spending usually happens via govts or wars whereas this time, it’s four companies racing to build the foundational mechanics of agi.
kinda insane that the next layer of civilization is being ~entirely privately financed before most govts even understand what’s being built. has this ever happened before?!
One aha from class with @alighodsi at MS&E 435 this wk:
Open source closing the gap with closed source may be inevitable.
Why? Distillation.
The old training substrate was the internet. Common Crawl is roughly 2T tokens (10^12).
The new training substrate is AI-generated output. If OpenAI + Anthropic have produced ~$50B of tokens at ~$5 per million tokens, that 10^16 of proprietary-model tokens in the wild.
That is four orders of magnitude more than Common Crawl. (10^4)
At some point, every training run is learning not just from humans, but from the exhaust of the best closed models.
The gap may close because *the teacher cannot stop teaching*.