Daniel Gibbs

Data Scientist - Stanford PhD interest and experience in R&D, biotech, climate change projects. RAG, NLP, Healthcare, Deep Learning, AI

[Daniel Gibbs @drbiomass]

#NSF #NIH #Stanford

[Adam Butler @GestaltU]

Fun fact: The 1998 paper that introduced Google and PageRank to the world ends with this acknowledgment:

"Supported by the National Science Foundation under Cooperative Agreement IRI-9411306. Funding also provided by DARPA and NASA."

Sergey Brin was on an NSF Graduate Fellowship. Larry Page was a PhD student on the grant.

Google—now worth $2 trillion—exists because American taxpayers funded "the Stanford Integrated Digital Library Project."

Not a startup garage myth. A government grant.

Every time someone says public research funding "picks winners and losers" or "crowds out private innovation," remember: the most dominant technology company of the 21st century was incubated entirely with public money, inside a public university, by researchers on federal fellowships and grants.

The private sector didn't see it coming. VCs passed. The government funded it anyway—not because it would become Google, but because fundamental research into information retrieval seemed worth understanding.

That's the point. You can't predict which grants will change the world. You fund the science and let researchers explore.

The internet (DARPA). GPS (DoD). Touchscreens (CIA/NSF). mRNA vaccines (NIH). Google (NSF/DARPA/NASA).

Public investment in basic research isn't wasteful spending. It's the seed corn of the entire modern economy.

[Dr. Angela Rasmussen @angie_rasmussen]

Two years ago, we demonstrated that the SARS-CoV-2 pandemic likely began at the Huanan Market via zoonosis resulting from the wildlife trade.

Many unanswered questions remained about the animals themselves.

Some answers, out today in @CellCellPress.

https://t.co/GwnUlfRjN9

[Daniel Gibbs @drbiomass]

#Nvidia #Groq Some analysts not getting the picture here. Nvidia plus Jonathan Ross et al. can mix LPU technology with Nvidia development at scale and create a whole new inference market share worth a lot more than $20B.

[Daniel Gibbs @drbiomass]

#Nvidia #Groq In effect, Nvidia plus Jonathan Ross have the potential to build a new type of fast low-latency inference chips, building on Groq’s hardware and software.

[Daniel Gibbs @drbiomass]

#Nvidia #Groq In effect, Nvidia plus Jonathan Ross have the potential to build a new type of fast low-latency inference chips, building on Groq’s hardware and software.

[mary davis @marydavis446257]

Nvidia is licensing Groq's AI chip tech & hiring its CEO Jonathan Ross! 💻🚀 Groq's LPU chip claims to run LLMs 10x faster & use 1/10th the energy. Game-changer for AI computing? 🤔

#Nvidia #Groq #AIChips #TechNews #HimalayaAtAWS #VeerBalDiwas #FridayFitness

[Daniel Gibbs @drbiomass]

#Nvidia #Groq

[Rihard Jarc @RihardJarc]

I just published my piece on $NVDA's $20B "acquisition" of Groq & its big importance:

- How is the Groq LPU different than GPU/TPU
- The strategic play from $NVDA
- What does this mean for $NVDA and the industry?

Bottlenecks, bottlenecks, bottlenecks...

https://t.co/4fSAwnSLYW

[Daniel Gibbs @drbiomass]

Great review of #Groq #Nvidia deal technology and business

[Ben Pouladian @benitoz]

’Twas the night before Christmas and all through AI

Everyone thought Groq was a TPU guy

It’s not

144-wide VLIW no DRAM fully deterministic
Nvidia didn’t buy a product

They studied an edge case and neutralized a future threat

Full breakdown 👇

https://t.co/eC1Fc7Aiq1

[The Nobel Prize @NobelPrize]

“Oh my god… I have no words.”

Listen to the emotional moment this year’s laureate Maria Corina Machado finds out she has been awarded the Nobel Peace Prize.

Kristian Berg Harpviken, Director of the Norwegian Nobel Institute, shared the news with her directly before it was announced to the world.

#NobelPrize #NobelPeacePrize

[Daniel Gibbs @drbiomass]

The one central observation is that animal brains reflect the body they inhabit. Since #AI models lack a body, they are vastly oversimplified compared to animal #brains. Add in survival and social behavior and learning, and there is a long way to go past #LLM evolution.

[Daniel Gibbs @drbiomass]

“Animals” are a highly diverse group, and this diversity is reflected in their brains and bodies, e.g. insect brains are largely devoted to olfactory and sensory processing, with most motor control delegated to ganglia further downstream.

[Daniel Gibbs @drbiomass]

“Animals” are a highly diverse group, and this diversity is reflected in their brains and bodies, e.g. insect brains are largely devoted to olfactory and sensory processing, with most motor control delegated to ganglia further downstream.

[Andrej Karpathy @karpathy]

Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.

As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough!

In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done".

As for my take...

First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone.

Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively.

I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise.

So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.

Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination.

[Daniel Gibbs @drbiomass]

#CUDA #GPU #LLM From Scratch #Substack

[ℏεsam @Hesamation]

I think I found a based Substack on low-level GPU programming by accident.

He has some extensive articles on CUDA programming, building LLM inference engines, looking inside GPUs and much more.

even the name is cool: "From Scratch". bro.

[Daniel Gibbs @drbiomass]

#China #AI papers

[Carlo Edoardo Ferraris @carloAI]

7/ China doesn’t see AI as a product. They see it as infrastructure.

That mindset shift changes everything.

Silicon Valley is optimizing for user growth.

Beijing is optimizing for national resilience.

This isn’t a startup game. It’s a civilization shift.

[JoeinNJfightsfordemocracy @progavalanche]

In May every past director of the National Weather Service signed a letter warning DOGE cuts to NOAA & NWS would cause needless loss of life.

We didn’t have to wait until after the midterms to see the impact of this Republican policy.

#DemVoice1 #wtpBLUE

[Sen. Bernie Sanders @SenSanders]

Speaker Johnson says there's “a moral component” to Trump's “big, beautiful” bill. Really?

Is it “moral” to cut nutrition assistance for hungry kids?

Is it “moral” to eliminate health care for 13.7 million Americans so the 1% gets a $664 billion tax break?

I don’t think so.

[Michael Lin, MD PhD 🧬 @michaelzlin]

Our NIH grant to discover coronavirus antiviral meds was terminated today.

With this grant, we had developed a better SARSCoV2 inhibitor than Paxlovid, and we recently discovered an improved drug that looks better than Pfizer's own second-generation inhibitor.

[Pete Muntean @petemuntean]

BREAKING: The FAA is halting flights into Miami, Fort Lauderdale, Palm Beach, and Orlando airports for "space launch debris" until at least 8 p.m. EST.

The move follows the failed launch of SpaceX's Starship, which lifted off from Starbase in South Texas about 45 minutes ago.

[Daniel Gibbs @drbiomass]

@DeryaTR_ It’s logical that primal amino acids in
asteroids would have equal amounts of ‘left-‘ and ‘right-handed’ forms. The abundance of left-handed forms in living organisms is probably due to natural selection during 4 billion years of evolution.

[The Tennessee Holler @TheTNHoller]

2 Economists:

1) tariffs act like a national sales tax, raising prices on consumers

2) Trump is using you to offset his tax cuts for the super rich, yet another transfer of wealth from the bottom to the top

[Daniel Gibbs @drbiomass]

#Tariffs cause #inflation and are effectively an undeclared sales #tax on consumers. Early warning from #Walmart. Not just food, but auto parts, tools, computers, clothing, school supplies.

[Joshua Reed Eakle 🗽 @JoshEakle]

Higher prices are coming.

Walmart is preparing to raise prices if Trump delivers on his tariffs promises.

Walmart CFO:

"We never want to raise prices... But there probably will be cases where prices will go up for consumers."

He added, “Tariffs, though, are inflationary for customers... [and they] certainly would add product costs."

Remember kids, foreign countries don't pay the tariffs.

You pay the tariffs.

[Daniel Gibbs @drbiomass]

#Deportation #agriculture #groceries #inflation

[honeybadgersmybitch @honeybadgertoU]

This is a must-see! 💥💥💥💥💥💥
A farmer talks about how mass deportations will hurt be devastating! 💥💥💥💥💥💥💥💥🗯💥💥💥💥💥💥

[Daniel Gibbs @drbiomass]

#opensource @huggingface #LLMs #Nvidia

[Philipp Schmid @_philschmid]

How can you deploy and scale open-source AI securely on your infrastructure? Introducing HUGS—an optimized, zero-configuration inference service by @huggingface that simplifies and accelerates the development of AI applications with open models for companies. 🤗

What is HUGS?
💡 Zero-configuration deployment reduces setup time from weeks to minutes.
🔧 Optimized for @nvidia and @AMD GPUs, and soon @awscloud Inferentia and @Google TPUs
🔒 Deploy within your infrastructure, keeping data and models secure.
📚 Supports popular open Models, including @AIatMeta Llama, @GoogleDeepMind Gemma, @MistralAI, @NousResearch Hermes, and more.
🔌 OpenAI-compatible API allows for easy integration with existing applications.
⚙️ Built on Hugging Face's open-source, like Text Generation Inference and Transformers.
🌐 Available through AWS Marketplace, @digitalocean, Hugging Face, and soon on GCP and Azure.
💰 Starting at $1/hour per container on cloud marketplaces, free on DigitalOcean (pay only for compute).
🛡️ Enterprise compliance and long-term support with SOC2 compliance.