Michael Garfield 💫

📚 advancing scenius at scale
🌎 @atlasresear_ch 🎙️📝 #humansontheloop
🦚 ex @sfiscience @longnow @mozilla
🏅 grantee @cosmos_inst @osvllc

[Michael Garfield 💫 @michaelgarfield]

Where to place your attention is without question a moral act. This is the understanding I am trying to help inject into the conversation around generative large language models.

The more people energize worst-case outcomes the more we guarantee them. If we can help people get enough distance from their knee-jerk fear response to reflect on their own minds and, as the bumper sticker says, not believe everything they think, we can interrupt the fear stream with an executive reallocation mandate that brings us rapidly and coherently closer to worlds we actually want to live in.

Silicon Valley is having a bad trip right now. It needs spiritual guidance if we are to live the promise of these tools.

[harley turan @hturan]

building a force-based alife sim with fable 5.2, including hunting, avoidance, and blended gene inheritance.

🎶 μ-ziq & mrs jynx — secret garden

[Michael Garfield 💫 @michaelgarfield]

@majamediaco @imbue_ai Pssssh you know this is the kind of discussion scheming I want to do and the kind of place I want to do it in

[Michael Garfield 💫 @michaelgarfield]

🎧 Why Can't AI Laugh at a Joke?

w/ @blamlab & @adamsafron on John Krakauer's recent papers with David Krakauer and @MelMitchell1 of @sfiscience on how we might find genuine signs of emergence in LLMs, and @quiltydunn on "the deliberation taboo" in cognitive science.

Full show notes & MANY more deep discussions on tech, science, and philosophy @ https://t.co/otTKNWUKau

Chapters:

00:00 Teaser & Intro
08:28 Meet John Krakauer
10:53 Against Internal Models
13:01 Replay and RL Critique
16:37 Mental (Not Neural) Representation
20:55 What Counts as Evidence
24:03 Why Humans Deliberate
26:56 #AI Mirrors the Taboo
28:42 Deliberation Taboo Framework
32:04 Chain of Thought Confabulation
34:30 Slow #Science and Creativity
38:15 #Cognition vs #Consciousness History
43:57 Pain and Conscious Advantage
46:09 Illusionism and @danieldennett
49:12 Working Memory Bottleneck Debate
51:03 @maxsbennett on Simulation Limits
53:19 Defining Real Emergence
56:38 Inside LLM Representations
57:56 NeuroAI Chick Experiments
01:04:08 Efficiency And Evolutionary Pressure
01:07:00 Laughter, Feelings, Deliberation
01:13:53 Basal, Cortical, & Conscious Intelligence
01:19:57 #WorldModels vs World Policies
01:23:27 Language’s Role & The Meaning Gap
01:27:54 Tacit Learning & One-Shot Rules
01:30:18 Admitting The Deliberation Taboo
01:31:12 Explanatory Gaps And Coarse Graining

[Michael Garfield 💫 @michaelgarfield]

With AI on everyone’s mind, we have to ask: what does it mean to think?

When does the brain actually create predictive models and rely on mental representations — and when is it really just a matter of inputs and outputs with no simulation going on in between?

Why do we feel pain and find things funny? Not just avoid damage or understand the logic of a joke, but actually feel these things?

Agreeing on what we don’t know matters, not least because trillions of dollars are invested in the dream of artificial general intelligence — but if thinking emerges in AI, how would we even know where to look? Behavioral indicators aren’t enough. What is the internal structure that would let us say with confidence “There’s real thinking going on here?”

Because as the success of deep neural networks makes clear, most intelligence doesn’t need this extra layer. Most intelligence isn’t thinking — at least, at the level of the whole organism or AI system. And, as much as we want to believe otherwise, maybe none of it has to be.

We need to remind ourselves what we don’t know. This is why guest co-host @adamsafron and I called on neurologist and neuroscientist John W. Krakauer, Director of the Johns Hopkins @blamlab and External Professor at the @sfiscience, who has spent decades studying the modularity of the brain and the relationship between sensorimotor functions and deliberative thought through research on stroke patients.

Our conversation with John gets at the very heart of the biggest taboo in cognitive science: that there is a difference between intelligence and thinking…that we’re conscious, and we don’t know why, and these distinctions matter to intelligence research. As AI demonstrates more and more capability to do things we thought it took conscious thought to achieve but shows no indisputable concrete evidence of being conscious, it turns out science still doesn’t understand what consciousness is.

Humans On The Loop Episode 41 is now everywhere you go for podcasts!

#cognitivescience #ArtificialIntelligence #neuroscience #sciencepodcast #consciousness

[Michael Garfield 💫 @michaelgarfield]

@tomowenmorgan Did you freewall one post or EVERY post?

[Michael Garfield 💫 @michaelgarfield]

@sopharicks @getjonwithit Uhhhh...More is Different, Phil Anderson 1972

[William from Restless Egg 🥚 @William_at_RE]

tech should provide people limit experiences, moments of rash exuberance, it should scale those in ways that life does not naturally do.

entertainment is obvious vehicle, but not just bcs a show is itself amazing but bcs so many people can feel pride: sound person, visuals, builder, artist, the person who brought friends

tech is fundamentally a coordination mechanism for limit experiences (it is not an office manager or productivity widget in essence)

and those experiences are what’s fundamentally human, moments of non-underrstandjng and going beyond the limits of knowledge. that’s where the renewable resource of spirituality lives

[Michael Garfield 💫 @michaelgarfield]

@knuthings Then again it's also increasingly trivial to falsify signatures

https://t.co/WlgTEniYr2

[Michael Garfield 💫 @michaelgarfield]

@owocki

[Michael Garfield 💫 @michaelgarfield]

Hey hey! Yes, changed the name of the show...might change it back now that Meta is using the name of my show in their press releases, LOL.

Not anti-AI, because I'm with Jaron Lanier that AI isn't even one thing. Co-founded an R&D group working toward a future Web grounded in living systems principles. Being anti-anything is probably unavoidable but also, negation tends to just amplify things so I'm pro doing AI right and that's not just a whole conversation, that's a whole show. Which is what's up now. Welcome back!

[Michael Garfield 💫 @michaelgarfield]

It's Fall 2026. If you aren't wondering whether Technocapital is destined to overrun everything and take over the universe (or burn out and leave, at best, the ruins of a Dyson sphere for the next post-singular civilization to dig up), you should be.

Luckily, Charles Stross gave us the perfect book to anchor deep conversation about this.

@krishnanrohit and I just met with a small group of friends to discuss Stross’ Hugo-winning 2005 novel Accelerando, a book that felt like a masterful exercise in future shock induction when I first read it in 2009 and (as Rohit put it) “a Wikipedia entry from the future” when I revisited it this fall. This is a story that doesn’t shy away from thinking about the unthinkable future, and offers us a deliciously skewed lens through which to think about our present day condition. Appropriately, this conversation was a blitz of speculation on the political, economic, and psychological dimensions of human life amidst an “intelligence explosion.”

The Big Idea in Accelerando is that self-organizing market-as-thermodynamic-gradient-descent processes, driven to extremes by automated finance, disrupt everything from memory and attention to governance to the organization of entire solar systems.

The question we asked from a dozen different angles was, “How flawed is this premise, exactly?” Is this prophecy, or does it miss something crucial about the evolution of complex systems? Does any sufficiently advanced complex system eventually evolve something like morality, or is an insatiable hunger for novelty the answer to Fermi’s Paradox?

Easily one of the wildest conversations we’ve had in a while. 👇🏼

[Michael Garfield 💫 @michaelgarfield]

@RichDecibels Are you saying we might all actually...wake up?

From your lips to God's ears. And by God's ears I mean every set of ears.

[Michael Garfield 💫 @michaelgarfield]

I really appreciate people using AI's affordances for endless variety and leaning into the "adaptive branding" future where it's not just the PRICES that are unique per customer

[Michael Garfield 💫 @michaelgarfield]

We covered a lot of ground — but left more than enough for a second call, which will happen before Thanksgiving. Stay tuned!

• How Accelerando handles the issues of information overproduction and dysregulated executive function (e.g., post-singularity intelligences ruining a democratic vote with an influx of reincarnated historical figures and fictional characters).

• How Manfred Macx’s “venture altruism” strategy — filing IP into a self-regulating structure of autonomous shell companies to participate in a gift economy while protecting work from malicious expropriation — inadvertently establishes a precedent for machine economies where humans can’t compete, creating zero-accountability entities capable of bypassing legal jurisdictions and resisting distributed denial-of-service attacks.

• How post-singularity civilizations face a voracious hunger for novelty to prevent model collapse…ultimately defaulting on entropy loans when unable to absorb sufficient new cognitive perspectives.

• How modern companies function as amoral super-intelligences capable of actions beyond individual human capacities, requiring complex external structures to manage negative externalities…what new regulatory systems have to emerge in order to hold organizational super-intelligences accountable?

• How characters like Sadeq and Pamela, who maintain their moral convictions and resist runaway digital immortality, model the restraint that would make post-singularity civilizations viable.

• Whether mediums of exchange like currency that act as mechanisms for the scaling of transactions inherently drive Goodhart’s Law optimizations that destroy their own substrates.

• How no matter how weird things get, they will always feel normal to our kids…

https://t.co/HCrRkOdPrD

[Michael Garfield 💫 @michaelgarfield]

Mainly, ignorant of the economy as a complex system.

In a fast-changing network, chokepoints get routed around, and tightly linked exposures make the system more fragile.

Tech *will* emerge route around tolls: open payment standards for AI agents, models running on users' own devices, decentralized training, and public payment systems are just a few.

[Michael Garfield 💫 @michaelgarfield]

Decided to share this here instead of elsewhere on the thread b/c it seems to fit better:

I feel like "context" isn't just "everything goes in a repo." Someone can relay a joke in exquisite detail, tell you everything they possibly can about the situation, and it's still not funny because "you had to be there" isn't just "you need more data" — and I believe this is what you are saying @kaseyklimes, and that we agree on a deep level. But I just squirm hearing "context" used in this way because it is often definitionally the invisible environment (e.g., norms) that confers meaning to whatever we decide to make legible. There is no such thing as total legibility. (Cataloguing every species isn't studying their food web and then you realize that and sample every genome in an ecosystem and still have no record of the geological/climate constraints, etc. Maps don't tend to show what *isn't* known or valued.)

Maybe I'm using to strong/restrictive a definition, but it's in response to this notion that with enough data we can "capture" context, when it's an endless regress we can easily chase over a cliff.

Context *is* king. But metrics inherently strip context. Like, always.

Okay, time to log off.

Related:
https://t.co/zfb5tSqlih

[Michael Garfield 💫 @michaelgarfield]

Here's why I'm placing my bets on nested sovereign infrastructure for high-dimensional collective foraging in an age when:

- We need plausible desirable futures
- AI is a self-fulfilling prophecy machine
- Reality debt compounds exponentially as we accelerate
- Context is king
- Only fools compete with the price of electricity

https://t.co/I7ZmmYcTzU

[Michael Garfield 💫 @michaelgarfield]

Where to place your attention is without question a moral act. This is the understanding I am trying to help inject into the conversation around generative large language models.

The more people energize worst-case outcomes the more we guarantee them. If we can help people get enough distance from their knee-jerk fear response to reflect on their own minds and, as the bumper sticker says, not believe everything they think, we can interrupt the fear stream with an executive reallocation mandate that brings us rapidly and coherently closer to worlds we actually want to live in.

Silicon Valley is having a bad trip right now. It needs spiritual guidance if we are to live the promise of these tools.

[Michael Garfield 💫 @michaelgarfield]

@citrini Would love to write it with you

[Michael Garfield 💫 @michaelgarfield]

@knuthings That's just ONE of the places this breaks down.