pavan kuppili

Eng Dir@Google. Tech. AI. Long India. Dad! Views my own.

[Lilian Weng @lilianweng]

A super long overdue (3+ years?) post on scaling laws.

Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.

The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.

https://t.co/HP26eJvjHB

[pavan kuppili @pavan_sk]

Growing up in India, the usual depiction of British colonial administrators was negative. But many of my family elders spoke of Sir Arthur Cotton in highly reverential terms. With his dams and irrigation projects, Sir Arthur Cotton permanently transformed the Andhra region and improved countless lives, and is revered till this day. Engineering and technology can be a force for enormous good across the world. https://t.co/ee7Q69dIPt

[N Chandrababu Naidu @ncbn]

గోదావరి జిల్లాలను సస్యశ్యామలం చేసి, అన్నదాతకు శాశ్వత సంపదను అందించిన మహనీయులు సర్ ఆర్థర్ కాటన్ గారిని ఆయన జయంతి సందర్భంగా స్మరించుకుంటున్నాను. కాటన్ తన దూరదృష్టితో గోదావరి డెల్టాకు జీవం పోసి లక్షలాది రైతు కుటుంబాలకు వెలుగునిచ్చారు. అందుకే ప్రజల గుండెల్లో ఇప్పటికీ కాటన్ సజీవంగా ఉన్నారు. రాష్ట్రంలో ప్రజా ప్రభుత్వం చేపట్టిన సాగునీటి ప్రాజెక్టుల నిర్మాణానికి, సమర్థ నీటి నిర్వహణకు కాటన్ దొరనే స్ఫూర్తి అని తెలుపుతూ నివాళులు అర్పిస్తున్నాను.

[pavan kuppili @pavan_sk]

@elonmusk 10 year old likes to identify fun causal links: Collapse of Roman empire -> Gibbon's book -> Asimov's Foundation -> Musk starting SpaceX -> internet on flights!

[pavan kuppili @pavan_sk]

Another world cup victory. Another equally incredible performance by Bumrah. 🐐🐐

[pavan kuppili @pavan_sk]

GOAT! Comes in with insanely high expectations, and delivers even more. 4.17 economy rate, while bowling at the toughest times 🤯 #Bumrah

[pavan kuppili @pavan_sk]

Back-to-back world cup victories, and utterly dominant in the final. Indian cricket is now a juggernaut! 🇮🇳🏆

[ICC @ICC]

𝐂𝐡𝐚𝐦𝐩𝐢𝐨𝐧𝐬 𝐨𝐟 𝐭𝐡𝐞 𝐰𝐨𝐫𝐥𝐝, 𝙤𝙣𝙘𝙚 𝙖𝙜𝙖𝙞𝙣 🇮🇳🏆

#T20WorldCup

[pavan kuppili @pavan_sk]

Lawrence-al-Gaib! One of my fave history books: https://t.co/6O6EL1XUoU

[Seth Frantzman @sfrantzman]

Dune, but it’s set in 1916 with a British officer sent to the Middle East

[pavan kuppili @pavan_sk]

Chinese robots doing martial arts 🤯

Can we solve laundry and dish washing now.

https://t.co/tpvt8QIc2e

[pavan kuppili @pavan_sk]

Human tools always shatter evolutionary speed limits.

It took eons to solve flight; then we went from the Wright Brothers to Apollo 11 in one lifetime.

Human brains took 2 million years to triple in size. AI compute doubles every few months.

The arc of intelligence explosion is clear. Our job is to channel it productively.

[pavan kuppili @pavan_sk]

Lol. Australia getting knocked out early (by Zim!) in a WC brings joy to and unites the rest of the cricketing world.

[Piers Morgan @piersmorgan]

My thoughts with @CricketAus at this very difficult time. 🙏

[pavan kuppili @pavan_sk]

What an addictive song. #Dhurandhar

https://t.co/qC9XWEuZgV

[Simon Willison @simonw]

I wrote about Clawdbot/Moltbot/OpenClaw and Moltbook, the fascinating, weird and sometimes even useful social network for digital assistants to swap tips and gossip with each other https://t.co/1udoHpxs5g

[pavan kuppili @pavan_sk]

@stratechery Another "Humans Want Humans" example: Computers might play "perfect" chess, but nobody cares. We want to see Magnus play with all the ego, nerves, drama, and human genius.

[pavan kuppili @pavan_sk]

⚡⚡⚡

[Google @Google]

Gemini 3 Flash is now available ⚡

Since introducing the Gemini 3 series last month, we've seen you vibe code simulations to learn about complex topics, build and design interactive websites and understand multimodal content. Now we're introducing Gemini 3 Flash, our latest model with frontier intelligence built for speed

Gemini 3 Flash is starting to roll out today as the default model in the @GeminiApp and in Search via AI Mode, and is also available in our developer and enterprise products.

[pavan kuppili @pavan_sk]

@architgupta "India after Gandhi" by R Guha is pretty good

[Sachin Tendulkar @sachin_rt]

1983 inspired an entire generation to dream big and chase those dreams. 🏏

Today, our Women’s Cricket Team has done something truly special. They have inspired countless young girls across the country to pick up a bat and ball, take the field and believe that they too can lift that trophy one day. 🏆

This is a defining moment in the journey of Indian women’s cricket.

Well done, Team India. You’ve made the whole nation proud. 🇮🇳 💙

[pavan kuppili @pavan_sk]

Great to see Google AI Hub in Vizag, with a gigawatt-scale compute capacity. #cityofdestiny :)

[Sundar Pichai @sundarpichai]

Great to speak with India PM @narendramodi @OfficialINDIAai to share our plans for the first-ever Google AI hub in Visakhapatnam, a landmark development.

This hub combines gigawatt-scale compute capacity, a new international subsea gateway, and large-scale energy infrastructure. Through it we will bring our industry-leading technology to enterprises and users in India, accelerating AI innovation and driving growth across the country.

[pavan kuppili @pavan_sk]

Interesting framing. LLMs as summoning ghosts vs building animal intelligence :)

[Andrej Karpathy @karpathy]

Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.

As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough!

In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done".

As for my take...

First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone.

Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively.

I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise.

So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.

Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination.

[pavan kuppili @pavan_sk]

"The single most important reason for the U.S. to own part of Intel, however, is the implicit promise that Intel Foundry is not going anywhere. There simply isn’t a credible way to make that promise without having skin in the game, and that is now the case."

[Stratechery @stratechery]

U.S. Intel

The U.S. taking an equity stake in Intel is a terrible idea; it also happens to be the least bad idea to make Intel Foundry viable.

https://t.co/8dcz5R9v7Y

[pavan kuppili @pavan_sk]

"Why Machines Learn" is a fun read. Also learned some new things! Didn't know about Hopfield networks which was a cool idea of building associative memory (ex. retrieiving original images from corrupted images) using a simple version of neural nets. @anilananth

[Anil Ananthaswamy @anilananth]

Sometime in Dec 2021, I got to talk to Ilya Sutskever, for my book WHY MACHINES LEARN (it was before the ChatGPT era; doubt I'd be able to do so now).

Ilya said this about the math of deep learning that he encountered in the first papers he read on the subject (given to him by Geoff Hinton): “How can it be that it’s so simple . . . so simple that you can explain it to high school students without too much effort?...I think that’s actually miraculous. This is also, to me, an indication that we are probably on the right track. [It can’t] be a coincidence that such simple concepts go so far.”

Even as he said this, I thought--and told Ilya--that this statement of his could be the prologue for Why Machines Learn!

There's indeed something about the math of ML--a feeling reinforced when I listened to Misha Belkin's recent Simon Institute talk: https://t.co/f02K2Hj4kn

Misha said: "Fundamentally, modern AI is just a mathematical object. Mathematics is transforming the world at a very fundamental level, especially with respect to modern AI...Mathematics is at the core of modern AI. There has never been a time when it was more important to understand mathematics."

(Misha was one of three people who read/reviewed WHY MACHINES LEARN from beginning to end. I owe him a huge debt.)

It's almost pub day for the paperback editions of WHY MACHINES LEARN (26 Aug in the US, 28 Aug in the UK and India). It's my ode to the math of ML.

[pavan kuppili @pavan_sk]

"You might see other cars where it looks as if one person's designed the front end of the car and someone else did the back end. If there's one thing I hope to be remembered for, it's the cars that I've been overall responsible for look cohesive" - Adrian Newey, F1 designer

Design cohesion is such an important principle across disciplines.