Cloclo

BD @Auros_global | Prev @ubs | 𝔭𝔯𝔞𝔵𝔦𝔰 𝔠𝔦𝔱𝔦𝔷𝔢𝔫

[Jensen Huang @JensenHuang]

https://t.co/pppEbPVtfE

[François Chollet @fchollet]

A significant portion of current AI "discourse" is less about technological capabilities and more about frontier lab employees navigating their own self-esteem and sense of identity.

[Jukan @jukan05]

TFHK Commentary: How Should We Understand the Current Correction in AI Hardware?

The market is always right. Changes in stock prices inevitably reflect the new variables the market is currently pricing in. Even as long-term bulls on the AI industry, we need to understand the core concerns driving this correction in AI semiconductor stocks.

The current market bears a striking resemblance to last autumn and winter. Following OpenAI’s large fundraising round, industry conditions were very strong, yet stocks continued to trade sideways. Market participants spent every day debating CapEx, ROI, valuations, and financing—much like they are doing now.

The conclusions from this quarter’s earnings reports remain overwhelmingly positive. GCP grew by 80%, the ROI of cloud investment was validated, Intel delivered a significant beat, and ASML, TSMC, and Intel raised their order or CapEx outlooks. Presumably, these companies also saw extremely strong downstream forecasts, giving even the most conservative players in the supply chain the confidence to make aggressive bets. Had this information emerged in May or June, semiconductor stocks would almost certainly have surged.

Now, however, every earnings release has instead become an opportunity for bears to reassess valuations and the long-term investment thesis. What we may be seeing is that the AI market is no longer in the “AI Summer” of May and June. The same positive developments now provide less support to share prices. Take GCP’s 80% growth as an example. Previously, the market’s first reaction would have been: “AI demand has exceeded expectations—the catalyst has arrived.” Now, the first response is: “So what? What about 2028? Can OpenAI become profitable? For how many more years can GPU prices keep rising? Margins are rising again, financing costs are increasing, and the entire CapEx thesis needs to be repriced.” In essence, the market has shifted from trading the growth of AI CapEx to trading its sustainability and ROI.

Market sentiment, as we perceive it, has already become extremely bearish. Even long-term bulls are beginning to question whether AI semiconductor stocks can continue to rise, and we are hearing almost no calls for new highs. When the market shifts from looking for further upside to searching for additional downside risks, it usually means that pessimistic expectations have already been largely priced in.

Nevertheless, we have no doubts about the fundamentals. We also believe that, ultimately, facts determine stock prices. So what would send these stocks higher again? Under the framework outlined above, additional capital-spending plans alone will no longer be enough to convince the bears. What is needed is validation of a new demand curve. The most powerful and direct catalyst would be the emergence of a blockbuster product. If “Coding 1.0” proved that AI can improve developer productivity, then “Coding 2.0” must demonstrate that AI agents can genuinely replace part of the software development process. Once new productivity use cases are validated, the market’s concerns about AI ROI may be redefined, and AI infrastructure spending will once again be viewed as “productivity investment” rather than a “cost.”

[shira @shiraeis]

One of the most controversial takes I have is that ppl do not fundamentally want a world without hierarchy - they simply want a hierarchy whose scoring function rewards the traits they possess & recodes their rivals’ advantages as moral defects.

If you destroy aristocracy, ppl will rank themselves by wealth. Get rid of wealth & they rank by education levels, taste, political purity, beauty, suffering, proximity to power, technical competence, or even conspicuous indifference to status all together.

Status determines who receives / “deserves” attention, forgiveness, romantic access, institutional credibility, & the right to write history. Ofc material resources matter, but humans will often sacrifice material welfare to avoid symbolic subordination.

This begs the question: whose traits will the next social order classify as admirable?

There’s the permanent overclass / underclass post-ASI world answer - that those who own / control / remain economically complementary to AI versus everyone else - but this is itself somewhat biased by limited perspective on how the world will receive ASI (and also an artifact of projecting today’s status logic forward).

ASI may rewrite the scoring function entirely. Intelligence becomes cheap and technical competence obsolete and wealth less meaningful in at least one potential outcome. Traits like beauty / social coordination / authenticity / biological and genetic rarity could become the new basis of rank. What’s uncertain and the determining factor here is which human qualities remain scarce enough to warrant reverence in an age of abundance.

Someone who says merit is above all else usually has a particular theory of merit in mind. Likewise, someone who says lived experience matters most is also proposing an epistemic hierarchy. Rebels think courage is best, bureaucrats procedural fluency, technologists intelligence, aristocrats lineage, and revolutionaries idealogical virtue. Each of the aforementioned imagines that their preferred ranking principle is not merely advantageous but JUST.

None of this is to say injustice is fake or imaginary or equality before the law is pointless or every moral conviction is secretly fraudulent. It simply means the aspiration to eliminate hierarchy is an incoherent delusion. The humane objective post ASI will be to prevent any one individual hierarchy from becoming total.

A free society should contain many partially independent status systems. Losing one game because you’re not maxxed out on the be all end all trait shouldn’t mean losing everything in life simultaneously.

One of AI’s deepest dangers imo is therefore scalarization.
Models will become extraordinarily good at inferring latent traits from user behavior / human interaction. The danger here is if employers / platforms / lenders / governments / social networks all begin consulting variations of the same hidden representation. Then 100s of local imperfect escapable reputations collapse into one interoperable estimate of human worth / value.

It’d result in something resembling totalitarianism but without an official dictator or doctrine, just a universally queried embedding.

The system would know that you are low-ranked before anyone could explain why. Every institution would independently reach the same conclusion because they were not, in fact, independent. Your past would generalize perfectly and permanently.

So I think pluralism is kind of a protection against a single ranking function acquiring enough resolution to govern the whole, but then again, that might be obvious.

[Stratechery @stratechery]

Who’s Afraid of Chinese Models?

Everyone is worried about Chinese models, but the frontier labs will be fine; we need to enable open U.S. alternatives.

https://t.co/Q5cbH229jj

[Ethan Mollick @emollick]

Very few people know the amount of useful work that the current models can do in Code/Codex/etc. with the right setup

This is not a "rah rah you are so early" post, this is a "AI companies are doing a really bad job explaining what their systems actually do in a clear way" post.

[will depue @willdepue]

A Stargate for Data

Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data.

At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger clusters and datasets, and received incredible new capabilities in return.

But this exponential scaling is bound to hit some limits. Oddly enough, compute has compounded fairly smoothly without limit, with trillions flowing into hypercluster buildout. Instead, we’re starting to hit the limits of an exponential demand for data. Gone are the days of being purely in the compute-limited regime, where we had effectively infinite internet data but never enough GPUs, we’re now entering a data-limited regime.

Luckily, this limitation is coinciding with staggering improvements in AI capabilities. Incredibly, we seem to have a real line of sight towards automating a majority of knowledge work with the methods we have today. RL + pretraining, and the data for each, will be generally sufficient to achieve most economically valuable tasks, given some minimal algorithmic progress and continued compute scaling.

In a data-limited world, economic progress & scientific acceleration will be directly bottlenecked by our coverage in each domain. We need to see data collection as imperative, deserving the same civilizational ambition we’ve given compute.

The internet as a one-time subsidy

It’s underrated how much all progress in AI owes everything to the blessing of the internet, this one-time civilizational subsidy to deep learning, decades of unintentional accumulation of a perfect dataset: every book, blog post, image, video, paper, discussion, etc. all digitized and freely available. Without the internet, we’d likely see comparably minimal progress in AI today, and in fact, if you notice where systems currently underperform, it’s almost always a domain where web coverage is limited and data is private, expensive, non-digitized, or non-existent.

But we’re running out of it. There are only about 300 trillion tokens of useful public human text, and the internet doesn’t produce nearly enough new high-quality data to match what scaling demands — we’re soon to hit the limits of public data for pretraining. And though the advent of RL bought us reprieve — chain-of-thought RL needed a new form of untapped data, gradable math & coding tasks, also available online — we’re quickly running dry of hard tasks for RL as well.

Why do we need so much data anyways? Humans learn comparably in far less time, needing just one textbook where language models might need the equivalent of hundreds to learn a new topic. It’s possible we discover methods that are massively more data efficient — synthetic data, data efficient architectures, other exotic algorithms — but fundamental progress is slow and highly unpredictable, and the recipe we have just works today.

And, while I’m wary of getting too deep here, even arbitrary data efficiency can’t replace data that just doesn’t exist in the first place. There’s a massive amount of missing information on the web: the dark matter of the internet — tacit knowledge, undocumented processes, etc. — most of which was never published and lives only inside organizations, the physical world, or just in people’s heads. I’ll leave it here and say, for reasons far longer than I can fit in this post [1], it’s best to operate on the assumption that our insatiable desire for data will continue as it has for the last decade.

There will be >$100B/year in data spend by 2030

We’re not screwed yet, of course. Only a fraction of useful data in the world is on the public internet, the rest is stored inside private datasets, corporations, personal archives, universities, governments, and otherwise. Labs can and will continue to license these private datasets, or create them from scratch, like Anthropic’s book scanning project. And we’ll increasingly task human experts to manufacture new high-quality data, with a large fraction of hard RL training tasks already being sourced this way.

But collecting this data, unlike before, will be expensive. As the free internet dries up and demand for data rises, we should see labs investing equally in data as compute, likely spending a significant fraction of their compute budgets on data. As we see trillions spent on compute, we should also expect hundreds of billions spent on data (human data & collection budgets), given their equivalent importance. And, notably, data spend is already tracking this way: total data spend across vendors, not counting internal lab efforts, is already roughly $7 billion per year. It’s quite reasonable we’ll see >10x by 2030.

Data is the moat

Data becoming increasingly private will also majorly shift the competitive landscape. While compute is a commodity — everyone buys the same chips and builds the same clusters — data really isn’t. The big reason why frontier models have felt eerily similar to one another, until now, is they were trained on substantially the same internet (pretraining data variability across labs seems pretty low). As labs diverge onto more exclusive, manually collected corpora, I think models will begin to increasingly diverge.

OpenAI pulling ahead in mathematics and Anthropic in cybersecurity isn’t an accident. I really think laser-focused collection of high-quality midtraining tokens, custom RL tasks, environments, with dedicated research effort, has driven much of the visible progress in the last year. James Betker has an excellent blog about “the ‘it’ in a model is the dataset”: model architecture and compute buy you efficiency and order-of-magnitude performance, but ultimately, models, of any architecture, are such incredible approximators of their dataset that the core meat of a model boils down to just that, nothing else. Data is a major moat.

AGI long, ASI short

As I’ve tweeted before, I’m confident that, despite the narrative, the data labeling industry will continue to fuel great businesses and be an excellent AGI long, ASI short. The argument is just: By the time the AGI labs no longer need data, it’s probably over for everything else too [2]. In this frame, the last companies left should be the data companies, as the last speck of economically relevant data is sucked in. And these companies are already among some of the fastest-growing companies in history: Mercor, founded three years ago, is rumored to be doing $2 billion in revenue with something like a few million expert labelers under contract.

While these businesses are very non-stationary, what type of data is needed shifts constantly, I don’t think that diminishes their value. The long-tail of the economy is long, and the value isn’t diminishing as you extend farther into more obscure information: as models get more capable, the value of the marginal dataset goes up, not down. Automating a full job means covering its full distribution of tasks, tools, edge-cases, and long-horizon loops. There’s some O-ring logic to it: a dataset that buys a 1% bump can justify a previously unjustifiable collection cost when it’s the difference between a system that does 99% of a job and one that does all of it [3].

The competitive dynamics of the data industry are still evolving but as demand for data is increasingly niche, ultra high-quality, expert-generated, I think we’ll see real consolidation. Again, contra-narrative, we’ll probably see true competitive differentiation built on brand, quality control of data (which, from personal experience, can vary massively), as well as in network effects from the talent networks themselves over time. We’ve already seen rapidly shifting data type demand work in favor of incumbents, benefiting those with early knowledge of where the market is headed.

The binding constraint

It’s truly remarkable that we seem to have the recipe — pretraining + RL — to absorb most economically valuable work, despite being far from a lot of what we expected from “AGI”. The same way chess engines revealed we never needed general intelligence to solve chess, as we originally thought, we’ll soon realize that software, mathematics, and the vast majority of the economy (including physical, just running ~3 years behind!) are the same. If recursive self-improvement or some other algorithmic breakthrough arrives, that’s wonderful, but we really don’t have to wait for it. The binding constraint between here and an automated economy isn’t that, it’s data coverage: every app, workflow, edge case, process, etc. sitting in private stores or someone’s head.

Ultimately, while we make tremendous strides in more efficient model architectures, and clusters like Stargate equip us with zettaflop-scale compute, we really aren’t making rapid progress collecting the data we lack.

We’ll soon live in a world where we have the methods & compute to accelerate scientific progress or economic growth, but not the data. And we’re already there today: frontier models would surely be as good at accounting/many medical tasks/legal advice as they are at software engineering if we only had the same pretraining & RL coverage as we did for code.

I really want to drill this in: The speed at which we automate the economy is going to be directly rate-limited by our ability to collect data about it.

Worth noting that under this assumption, with data as defensible and directly proportional to economic & scientific progress, data should also be considered a national strategic asset like compute. Imagine what we’d do in a world where we had a Manhattan Project-effort for AI and needed to mobilize data collection as a limiting factor. We should be concerned about China, with greater state capacity and authoritarian economic control, being capable of mobilizing data collection at national scale, potentially compounding their economy and scientific output faster than us down the line.

A Stargate for data

I’m leaving my complete ideas for a future post, as this one is already far too long, so I’d really like to pose the question here. Stargate exists because we organized trillions of dollars, international strategy, gigawatts around compute as a fundamental ingredient. What would equivalent ambition look like for data?

Obviously, scaling data collection, a heterogeneous mass of information across the economy, isn’t going to be as clear as scaling compute, as a homogenous infrastructural effort. A core division will be first, coverage — all uncaptured knowledge sitting across the economy/science/physical world and all that simply isn’t recorded — and, secondly, sheer volume in the domains we already train on: more hard math tasks, more high-quality web text, way more coding data, more legal drafts, etc.

I have a post coming soon which breaks down my proposals. There’s a lot of room for creativity. Quickly, we’ll probably want to start with a deep census of what we have and what we’re missing, predict what the 2030 model will still be bad at and work backward to what we should be collecting today. You can probably license a large amount, leveraging high lab valuations to buy datasets or companies altogether. There’s an adversarial nature to a lot of this collection with firms, so there’s lots of engineering to do this correctly. We should go convince important companies to turn off deletion policies, even if we’re not buying from them yet. Data flywheels in consumer products will be massive. Confidential training, government legislation for grant-funded research, running companies at a loss for their data, etc.

We’re headed towards hundreds of billions in expenditure, national prioritization, and major data limitation on the horizon. We have a great opportunity to think creatively about what a megaproject for data would look like: How do we, deliberately this time, construct the next internet’s worth of data?

Footnotes:

[1]: I’ll probably soon publish my much longer post explaining my position on data efficiency and why the value of this data is still pretty high in most worlds regardless of new algorithms.

[2]: The “AGI freeroll” bet: heads you win, tails ASI flips the world upside down anyways.

[3]: We already see a glint of validation of this point, given the data market is strongly tilting towards ultra-high-quality agentic data, rather than unskilled labeling — niche expert workflows, live environments, and evaluations requiring increasingly obscure talent & knowledge — yet shows increasing, not decreasing, revenues.

[Beff (e/acc) @beffjezos]

Token-maxxing with Fable eating up all your weekly limits while Sonnet subagents run in the background

[Etched @Etched]

We're coming out of stealth.

We've built our first racks after a successful A0 tapeout, $1B+ in customer contracts, and $800m raised.

Early customer tests show us achieving SOTA throughput, latency, and power efficiency on inference workloads.

Our first racks ship this summer.

[Dwarkesh Patel @dwarkesh_sp]

Always so much fun to chat with @3blue1brown

AI has been making much faster progress in math than in other fields.

As a result, mathematics is showing us, very concretely, what AI progress in other fields will look like.

Even within mathematics, there's a jagged landscape. What does it look like?

What is the nature of the most important conceptual breakthroughs in the history of mathematics, and how different are they from what AIs are currently able to do?

Does AI (on net) increase or decrease human understanding of the field?

How big is the overhang from having AIs systematically try to connect ideas already in the literature?

And what advice does Grant have for aspiring mathematicians, coders, and other students who are passionate about fields that are being most transformed upon by AI?

0:00:00 – AI is discovering new proofs. Is that AGI?
0:11:32 – The verification loop on conceptual breakthroughs can be a century long
0:26:12 – Will we understand an AI proof of the Riemann hypothesis?
0:38:08 – Can AI find the hidden bridges between fields?
0:53:48 – Why real-world tasks don’t fit into RL environments
1:07:07 – Good writing requires theory of mind that AI still lacks
1:16:02 – Why learning will still depend on human curation

Look up Dwarkesh Podcast on Spotify, Apple Podcasts, YouTube, etc.

[Science Corporation @ScienceCorp_]

The blade spins at 30,000 RPM through a wafer of exploratory biohybrid neural probes. They survive only if the water moves faster than the heat.

[Dwarkesh Patel @dwarkesh_sp]

What does the next training paradigm look like?

0:00:00 – The big research bet the labs are making
0:02:12 – Grindability is just as important as verifiability
0:06:10 – Will RLVR alone generalize?
0:08:41 – Getting the learning back to the weights
0:15:22 – Dreaming
0:17:23 – What 2027 looks like

Also on YouTube, pod feed, and Substack.

[Nick Grossman @nickgrossman]

https://t.co/GBy2md7JxD

[Cloclo @Cloclo77T]

Anthropic's Safety Superpower https://t.co/LtuKimwYGt via @stratechery

[TBPN @tbpn]

"We're about to see the explosion of analog."

@garyvee wants to open a restaurant that makes you check your phone in at the door and seats you at communal tables.

"Extreme AI is creating extreme analog. I think it's a barbell."

"I could not be more interested in physical retail, event-driven businesses, in concerts and venues."

"There are a lot of interesting non-digital realities that are coming as a countermove to the insanity of AI advancements."

"We're literally within a half decade of not believing a single video that's on the internet. In 5 years, if we're having this interview, most of the audience is trying to figure out if we're real or not."

"That is very real, and has substantial counter-opportunities."

"Any real entrepreneur, they're not crying about AI killing them. They're curious about how AI at scale is going to create opportunity for them."

[Arjun Khemani @arjunkhemani]

.@friedberg: “We are now at a moment where we are saying there is no longer private property in the United States.

This is one of the foundational rights that the founders of the United States tried to create: a distinction between these other nations that everyone flees from, where a monarchy or a totalitarian government or some communist system says everyone owns everything together, or some small number of people own and control everything.

And that’s what this always comes down to.

Whether it’s a socialist state or a communist state or a monarchy or some other totalitarian regime, there’s a small number of people that own and control everything.

And that is the brink that we’re on.

Because they are trying to say, for the first time ever, there is no longer private property in the United States.

That if the government can say everything that you’ve already paid your income tax on, and then you’ve bought and you now own, the government can take a piece of it every year based on the vote and the budgetary needs of an irresponsible fiscal legislature.

We’ve lost it all.

And that’s where we are.

And we see this just passed in Illinois.

People think it’s just crypto, just like they think that billionaire tax is just billionaires.

But anytime the government can take your private property after you’ve paid your taxes, bought something, and put it in your garage, we are done for.

That is when the politburo has unlimited capacity to tax and take and do what they want.

That’s the moment we’re at.”

[Rodolfo Quintana @rodolfoquintana]

How will you overcome the fundamental ultrasound tradeoff between spatial resolution and tissue penetration?

Higher frequencies improve resolution but attenuate quickly, while lower frequencies penetrate deeper tissue but reduce anatomical detail.

Is the scanner relying mainly on lower-frequency transmission tomography, AI reconstruction, or some new acoustic approach that changes this limitation?

[roon @tszzl]

transhumanism is an interesting side quest but 'the substrate is wrong' for the human/computer hybrid to be competitive with machine intelligence on feats of intellect. you have this low tech meat brain in the middle of all this lightspeed machinery, doing what exactly?

[Midjourney @midjourney]

Announcing a new division of Midjourney called "Midjourney Medical"

[TBPN @tbpn]

"I wonder if AI is contributing to contagion and mimesis in a way we don't fully understand." — @lukeburgis

"@tferriss was talking about how he had an 80% drop in sales of his books. And it coincided with 2022, when ChatGPT came out."

"He was asking why somebody would read prescriptive, how-to nonfiction, when they can go to ChatGPT and say, 'Here's my life, here are all the things that are going on for me. What should I do? And by the way, summarize Tim Ferriss' book, but make it highly personalized for me.'"

"Why would you read the book? You get instantaneous, highly personalized advice. I wonder, though, if that is actually going to lead to a form of contagion that we don't understand."

"There's something happening in the AI — even the engineers don't fully understand what it's doing — and it's giving us back something that feels really personalized to us. But the inputs are obviously being drawn by what LLMs and other people are putting into it."

"I wonder if we're entering a Pluribus (the Apple show) kind of situation, where the AI, while seeming highly personalized, is actually contributing to some form of social contagion."