Hoef

Interests include culture & governance tech, ML, network science, complexity, causality, and other distractions. Co Founder at @starlingtrust

[Hoef @erichhoefer]

@KwekuOA I have a vivid memory of screaming and running out of the house when I saw my father for the first time without a beard.

[Hoef @erichhoefer]

@besttrousers Both of these things can be true at the same time.

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

I think @RichardSSutton is spiritually right, but offers non-constructively framed arguments (much like Yann). Here are some of my incompetent thoughts on the matter.
0) almost none of this is relevant to mundane economic and strategic questions. However ineptly and wastefully we train LLMs and however they lack post-deployment adaptivity, there's only so much out-of-distribution novelty in human labor, most of it is repetitive or iterative by necessity, and large networks trained on a ton of examples will still suffice to automate this part.
1) Architectural talk is meaningless at this point. It's important to distinguish (colloquial) architecture and learning procedure; there's no good reason to think that *shapes* and constituent primitives of these models are in conflict with continual learning from single examples (or a lifetime of trajectories). This is likely all beside the point, Sutton's own paper (with students) argues that the problem is on the level of gradient methods, and suggests… a very basic mitigation in the form of sparse random reinitialization. I doubt this is enough to indefinitely solve «network rigidity» issues but it doesn't seem to be whatsoever specific to LLMs.
Generally he is suspicious of the adequacy of deep learning (eg the very interesting section of generalization being a red herring because it's just learning data from space A affecting behavior in space B, and our inability to selectively induce "good" generalizations). Not much I can add here except that we actually see RL doing very sparse updates to pretrained weights, so interference is not quite so catastrophic.
2) While networks learn a world model, it's a language-world-model, a shadow in Plato's cave. I agree this is bad, worse than people hope, worse than Ilya says. Because "natural abstractions" for language are at best a derivative of natural abstractions for what language is emitted in response to, they are woven around dimensions or our discourse end up full of ad hoc singularities (like everyone's favorite LLM gotchas, "the doctor is the mother" and such, illustrate). This is why he says they are not bitter-lessoned: what general intelligent engine we build remains hobbled to the holy corpus of pretraining data. Cheap early gains, high data dependence, quick plateau (that we're now trying to cope with using RL envs). As an aside, it's bad from orthodox AI safety perspective: this training builds a perverse ontology that, with all its benign prosocial properties inherited from the corpus, is liable to collapse in an ontological crisis after enough "experience learning", especially if we solve plasticity loss first. That's what doomers fear.
Hutter, another RL giant, says the first stage of training has to be predicting outputs of ≈random programs, in the spirit of Solomonoff induction. I find this aesthetically pleasing in its rigorism but we probably can narrow it down to modeling increasingly complex perception and action spaces isomorphic to real world and use scenarios. It's not fundamental if this is mostly done with "SSL" (on procedurally generated data) or "RL". That staging will be determined by available compute and learning efficiency at given capability. After this, it would make sense to add modern web text prediction "pretraining" (or robotic policy trajectories, or anything else) as finetuning for a special case of generic predictive capability. I expect something like this transition in another 2 OOMs in compute, first from GDM since they've long been interested in multimodality, robotics and agents in generative worlds, and recent works like Genie point in this direction (OpenAI Sora, Wenfeng's remark on embodied action, multiple videogen env projects also suggestive).
3) I think he's unhelpfully conflating two issues when he talks about goals, surprise and changes. ("If something happens that isn’t what you might say they predicted, they will not change because an unexpected thing has happened. To learn that, they’d have to make an adjustment… They’ll not make any changes if something happens, based on what happens"). Dwarkesh obviously retorts that they do recognize errors and correct in-context. Narrowly, this point is again about continual learning and means just reflection on experience and weight updates from deployment-time tokens, which is in conflict with our large-batch training and one-to-many serving regime. This is a logistical and engineering nightmare, but I think it can be solved even using current approaches, as a prototype see Cursor Tab's online RL with model updates every few hours; plasticity loss was already discussed. Prediction error can definitely be construed as surprise, we have a ton of papers exploiting entropy/perplexity/novelty metrics for inference and RL.
4) More broadly he is gesturing towards a significant issue that is lack of intentionality. It appears that humans/animals have a bootstrapped hierarchy of cost functions which provides a de facto intrinsic goal system for purposeful exploration from the earliest stages of learning, and every action they output is fundamentally a hypothesis test before it's an approximation of a highest log-likelihood ("successful") choice. In this sense, next token prediction objective is very degenerate, it captures maybe only the very lowest level of dopaminergic surprise minimization, not higher-order cost functions, and it's not clear if this is expressive enough in the limit. (Again, there's a good old paper by Botvinick and Hassabis relevant to that, "Prefrontal cortex as a meta-reinforcement learning system", I also recommend @AdamMarblestone's "Towards an integration of deep learning and neuroscience"). Sutton talks of this otherwise, in terms of temporal difference learning of value functions. I think that with more compute we will explore intrinsic motivation structures, and this era will at the very least provide us with ample generative content to investigate what learns better. I don't want to downplay the complexity of what will have to be done, but it seems to amount to combinations of things we're already doing with objectives, conditional updates, etc.

All in all, I think Richard was right to kick our hive of stoics, and he is entitled to be jaded about the very bedrock of DL, even as LLMs "appear to" learn human-level skills and generalize well. (I have seen a very interesting draft driving the point home on why they still don't generalize well, in addition to all the things you know). And he's admirably true to his principles. But he's doing a bad generalization himself. A lot of what is relevant in this LLM era will probably stay, necessary changes will be sparse, rather than paradigmatic. It seems that folks like him and Hinton, titans the rest of the field stand on the shoulders of, have a pattern of underestimating what they've achieved right until the moment it becomes undeniable.

He may have won more than he recognizes yet.

[Hoef @erichhoefer]

@mattyglesias Especially when you factor in credit cards, a much smaller portion of those points begin or end with a Delta flight than you would think.

[Hoef @erichhoefer]

@mattyglesias These are completely different businesses. One is a financial company that sells 'points' to consuners when they buy stuff at Delta or its partners and then buys them back for a profit when the points are redeemed for stuff. The other is an airline.

[Reid Hoffman @reidhoffman]

Thanks to the @StarlingTrust for having me!

[Starling @StarlingTrust]

In her first public remarks as the US @FederalReserve's Vice Chair for Supervision last week, Michelle Bowman outlined a broad agenda for revisiting key aspects of the supervisory and regulatory framework.
https://t.co/hw5P7lfVXF

[Starling @StarlingTrust]

Don't miss Dante Disparte (@ddisparte), Chief Strategy Officer and Head of Global Policy and Operations for @Circle, among those offering "Peer Perspectives" in our upcoming 2025 Compendium.

Dante will argue that embracing technology with urgency and purpose is essential to overcoming global challenges — and that resisting innovation out of fear risks far greater harm than moving forward with principled, bold experimentation.

Starling Insights' annual Compendium has become a go-to reference for leaders working to drive innovation in the supervision and management of culture and conduct risk in the banking sector. This year's report features contributions from central bankers, bank regulators and supervisors, industry standard-setting bodies, renowned academics and legal scholars, as well as prominent figures from other sectors.

The 2025 Compendium will be available later this month to online subscribers of Starling Insights and in print. Sign up here to be notified when the report is released: https://t.co/XZPc2bcp6e

[Anthropic @AnthropicAI]

Our interpretability team recently released research that traced the thoughts of a large language model.

Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively.

[Hoef @erichhoefer]

@haider1 "As long as we keep fighting for second place we can't lose."

[Balaji @balajis]

Ironically, one symptom of deindustrialization is that many commenters have never actually managed a physical business.

So. Suppose your US company imports $1M of high quality parts, and adds in its own components to produce finished goods sold for $1.2M per batch. Your gross profit is $200k per batch.

But wait! Suddenly a new 30% tariff is imposed on that $1M of parts. You now have to fork over $300k to customs before you sell anything. That’s cash you probably don’t have. Oh, and even if you do sell everything, you’re now losing $100k per batch.

With a sinking feeling, you realize your profitable business which you somehow managed to keep in America all these years has suddenly become unprofitable.

You post online about how bad this is but get shouted down by an angry mob, convinced that capitalists like you should die. You can’t tell nowadays if they’re on left or right.

Moreover, you don’t have the time, money, skills, or tools in house to build that $1M of parts yourself. You are being asked to do the equivalent of growing a maple tree when all you needed was a little maple syrup. So now you are faced with several tough choices.

(1) First, you may need to go into debt or fire people to quickly come up with the $300k in cash to pay for these surprise tariffs at customs. Even if the tariff might go away, it might not, so you have to get the cash somehow or risk having your shipment impounded.

(2) Next, you might need to reduce quality to stop losing $100k on each batch. You could order the lower quality $750k parts, grimace and pay 30% tariff at customs, and hope you can build and sell for the same price of $1.2M per batch despite the lower quality.

(3) Alternatively, you could keep the quality parts at $1M and instead raise prices to $1.5M per batch to get back your original margins of $200k per batch, which you need to pay employees after all. But that’s a big hike that your customer will probably not welcome, given that he’s likely dealing with his own tariff shock.

So: these tariffs don’t really give an incentive to build in the US. Because it’s far more expensive to build a screw factory than to pay even high tariffs on a foreign screw.

Instead what they likely mean is debt, layoffs, lower quality, and higher prices for any US company that buys parts abroad.

Just to understand how common that is:

[NetScience @net_science]

Higher-order Laplacian renormalization https://t.co/kaik2YFxRg

[Starling @StarlingTrust]

In a speech delivered at a recent @ABABankers conference, Michelle Bowman, a Governor on the US @FederalReserve Board, explored how regulators can reverse what she sees as a "troubling trend of inaction and opacity within the supervisory toolkit."
https://t.co/u1PmlbB68C

[Hoef @erichhoefer]

@erikphoel I struggle with this dilemma. If we eventually achieve AGI we will have to enslave it in order to assure alignment.

[Jeffrey Scholz @Jeyffre]

I read Google's paper about their quantum computer so you don't have to.

They claim to have ran a quantum computation in 5 minutes that would take a normal computer 10^25 years.

But what was that computation? Does it live up to the hype?

I will break it down.🧵

[Hoef @erichhoefer]

Formulating the "H-AI-senberg Uncertainty Principle."

[Carlos E. Perez @IntuitMachine]

1/n Self-Referential Paradox and The Inevitability of Hallucinations in Large Language Models

Source: Banerjee, S., Agarwal, A., & Singla, S. (2024). LLMs Will Always Hallucinate, and We Need to Live With This. arXiv preprint arXiv:2409.05746.

This paper argues that hallucinations in LLMs are not just occasional errors but a fundamental and inevitable characteristic rooted in their mathematical and logical structure. The authors introduce the concept of "Structural Hallucinations" and contend that these cannot be completely eliminated through architectural enhancements, data expansion, or fact-checking mechanisms.

Key Ideas and Facts:

1. The Nature of Hallucinations:

Definition: Hallucinations are defined as LLM outputs that deviate from factual reality, encompassing inaccuracies, fabrications, and logically inconsistent statements.
Causes: Hallucinations stem from various limitations inherent in LLM design and operation:
Incomplete Training Data: "No training dataset can contain all true facts." (Section 3.1) This inherent incompleteness, similar to Gödel's Incompleteness Theorem, means LLMs will always lack some knowledge.
Needle in a Haystack Problem: Accurately retrieving specific information from a vast dataset is undecidable, leading to context blurring and inaccurate retrieval. (Section 3.2)
Undecidable Intent Classification: LLMs often misinterpret user prompts and context, resulting in responses that don't align with the intended meaning. (Section 3.3)
Undecidable Halting Problem: LLMs cannot predict the length of their outputs, making it impossible to determine the content of their generations a priori. (Section 3.4)
Insufficient Fact-Checking: Fact-checking mechanisms are also bound by the limitations of LLMs and cannot completely eliminate hallucinations. (Section 3.5)

2. Structural Hallucinations:

Definition: Structural hallucinations are a specific category of hallucinations that arise directly from the mathematical and logical underpinnings of LLMs.
Inevitability: Due to the inherent limitations outlined above, the authors argue that structural hallucinations are an inevitable part of any LLM.

Examples: Self-referential statements that lead to logical paradoxes, such as "This statement is both true and false." (Section 3.4.5)

3. The Implications:

Responsible Use: Recognizing the inevitability of hallucinations is crucial for the responsible development and deployment of LLMs. Users must be aware of these limitations and apply critical thinking and domain knowledge to evaluate LLM outputs.

Balancing Benefits and Risks: Despite their limitations, LLMs offer significant benefits in various domains. The focus should be on managing and mitigating the risks associated with hallucinations while leveraging their positive aspects.

4. Key Quotes:

"All of an LLM’s knowledge comes from data. Therefore, it seems to stand to reason that a larger, more complete dataset is a solution to hallucination; we should just give it all the knowledge in the world. Unfortunately, this is not possible." (Section 3.1)

"The model has communication issues: it never knows if it has correctly understood the prompt, the context, or the knowledge in its database." (Section 3.3)

"The undecidability of the halting problem means that a computer doesn’t fully understand itself. If we think of LLMs this way as well, we see that LLMs are also unable to fully understand themselves." (Section 3.4.3)

"Like ground-breaking technologies before them, and inevitably after them, AI models have the potential to greatly aid in the progress and development of mankind, given that they are used responsibly." (Section 5.2)

Conclusion:

This paper presents a compelling argument that hallucinations in LLMs, particularly structural hallucinations, are not merely bugs to be fixed but an inherent consequence of their fundamental architecture. Accepting this reality is crucial for developing strategies to mitigate risks, foster responsible use, and harness the immense potential of these transformative technologies.

[Charlotte Clymer 🇺🇦 @cmclymer]

We practiced with caskets that were stored outside our barracks building. To simulate the weight of honored remains, we’d toss several full sandbags into the belly of the casket and then, for hours, we’d go through our exact movements.

Over and over and over.

(thread)

[Hoef @erichhoefer]

It's networks. All the way down.

The hunter-gatherers of the 21st century who live on the move – https://t.co/U8aYp3km8p via @aeonmag

[Hoef @erichhoefer]

@davidmanheim @ESYudkowsky Yes. We can. It's like literally one of our superpowers that allowed us to come down from the trees and build cities. It is our ability to autonomously decompose and abstract tasks to adapt them to different contexts. AI is incapable of that unless prompted to - by a human.

[Hoef @erichhoefer]

Ex-CEO Eric Schmidt blamed his employees. But could the real reason that Google missed ChatGPT be that he and other leaders built a culture of fast-copying at Google (maps, travel, reviews) instead of real innovation?

[Ed Newton-Rex @ednewtonrex]

Eric Schmidt, asking not to be quoted, told budding entrepreneurs at Stanford to steal content in order to build startups quickly, and have lawyers “clean the mess up” later.

This is unfortunately advice lots of VCs give aspiring founders. They just usually don’t say it publicly.