Ezra

[banteg @banteg]

>be me
>discover effective altruism
>apparently normal charity is inefficient
>why donate to random sad thing when spreadsheet can tell you optimal sad thing
>fair enough
>buy mosquito nets
>save lives
>numbers look good
>feel powerful

>couple years later
>someone asks an innocent question
>why only count people alive today
>huh
>future people matter too
>obviously
>my grandchildren shouldn't matter less just because they haven't spawned yet
>reasonable.jpg

>keep following logic
>what about their grandchildren
>also yes
>what about people in 500 years
>sure
>5000 years
>why not
>500 million years
>starting to get weird but morality is morality

>open calculator
>humanity could survive for an astronomically long time
>could colonize galaxy
>could have trillions upon trillions of descendants
>maybe digital people too
>maybe simulated civilizations
>maybe dyson spheres full of happy uploaded minds
>calculator starts smoking

>realize currently living humans are rounding error
>8 billion people suddenly looking extremely beta
>future contains potentially 10^something people
>can't even fit beneficiaries in google sheets

>new moral priority unlocked
>protect the long-term future

>stop thinking in units of "people helped"
>start thinking in "fraction of cosmic endowment preserved"

>malaria?
>terrible
>but only kills existing humans
>AI extinction could delete the entire light cone
>nuclear war could permanently derail civilization
>bad institutions could lock in terrible values for ten million years
>someone invents wrong constitution in 2140
>quadrillions suffer
>better fund governance workshop now

>friend says maybe we should improve hospitals
>explain opportunity cost
>friend says hospitals are full of actual sick people
>explain scope sensitivity
>friend stops inviting me to dinner

>need to decide what to fund
>easy
>expected value

>suppose project has one in a million chance of preventing extinction
>sounds tiny
>but extinction destroys 10^50 future lives
>multiply
>mother of god

>$10 million project has expected value of several galaxies
>charity evaluation complete

>someone asks where the one-in-a-million number came from
>expert judgement
>which expert
>us
>how calibrated
>extremely thoughtfully

>reduce estimate to one in ten million to be conservative
>still beats curing cancer by 38 orders of magnitude
>epistemic robustness achieved

>someone says maybe project doesn't work
>assign 20% chance
>still astronomical

>maybe project makes problem worse
>assign 5% chance
>still astronomical

>why 5
>because 30 felt pessimistic

>publish 46-page report
>contains seventeen sensitivity analyses
>every sensitivity analysis begins after assuming intervention has positive sign

>critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities
>yes
>that's literally why it's important

>critic says the uncertainty might be structural rather than numerical
>make probability smaller
>critic says no, I mean maybe your model is wrong
>make probability smaller again
>critic begins rubbing temples

>discover AI safety
>perfect longtermist cause
>AI might kill everyone
>or create utopia
>or seize galaxy
>or tile universe with paperclips
>or create billions of conscious software minds
>finally a problem with numbers big enough for me

>start AI safety nonprofit
>mission: prevent dangerous AI
>hire smartest people available
>smartest people immediately start building better AI to understand dangerous AI
>interesting

>we must understand capabilities to understand safety
>we must scale models to study alignment
>we must race ahead so less responsible actors don't get there first
>we must deploy systems to learn how deployment can go wrong
>we must build the thing quickly because building the thing quickly is dangerous

>outsider asks why the people most worried about AI apocalypse all work at AI companies
>complicated field

>company releases stronger model
>very concerned
>company begins training even stronger model
>extremely concerned
>company raises $14 billion
>concern reaches unprecedented levels

>need to influence government
>future is at stake
>normal democratic process too slow
>politicians don't understand exponential curves
>public doesn't understand x-risk
>experts must guide them
>who counts as expert
>people who understand x-risk
>who understands x-risk
>our friends

>someone objects that this seems politically convenient
>explain we're representing future generations
>future generations unavailable for comment

>develop concept of value lock-in
>terrifying possibility that one ideology controls civilization forever
>therefore extremely important that civilization adopts correct values before lock-in
>whose values
>let's circle back

>begin with impartial morality
>end with small group of people deciding what quadrillions of hypothetical beings would want
>beautiful arc

>meanwhile actual humans keep doing annoying things
>voting wrong
>having parochial attachments
>loving family more than strangers
>caring about local community
>getting upset when told their suffering is cosmically negligible
>evolutionary biases everywhere

>explain that moral intuition cannot be trusted
>except intuition that future digital people count
>and intuition that extinction is uniquely bad
>and intuition that our probability estimates are sane
>and intuition that our institutional choices improve the future
>those intuitions survived peer review

>someone donates $5k to local homeless shelter
>inefficient
>could have funded 0.0000000000003% of an AI governance researcher
>think of all the simulated people you just killed

>okay maybe don't phrase it that way publicly

>PR team says "future generations deserve a voice"
>much better

>journalist asks what longtermism means
>say "future people matter"
>everyone agrees
>great
>journalist asks what follows from that
>well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
>journalist raises eyebrow
>return to "future people matter"

>motte has entered the chat

>critic: of course future people matter
>me: glad we agree
>critic: I don't agree that your institute knows how to help them
>me: why do you hate our grandchildren

>eventually notice uncomfortable implication
>if future value dominates everything
>then helping people today mostly matters through effects on future
>education matters because future institutions
>health matters because future productivity
>democracy matters because future trajectory
>human beings slowly become instrumental variables in their own moral philosophy

>see starving child
>feel compassion
>check spreadsheet
>child's direct welfare contribution negligible
>but perhaps childhood nutrition improves national institutional quality
>compassion restored

>tell myself this is impartial altruism

>one day assistant asks obvious question
>"how do you know your intervention actually improves the far future?"

>silence

>open spreadsheet
>increase column width
>add confidence interval

>assistant asks again
>"no, I mean how do you know the sign is positive?"

>stare into cosmic light cone

>10^50 people staring back

>none of them exist
>none of them can tell me
>none of them can falsify my assumptions

>realize I have invented the perfect constituency

>infinitely important
>completely silent
>and always represented by me

[Ezra @Revan]

I don't know who needs to hear it but a tech ceo winning a power struggle for moral authority over the State would not actually be good and it's much better for the long term stability of civilization that we not open that particular can of worms for as long as possible.

[NIK @ns123abc]

🚨BREAKING: Pentagon is now calling Claude a threat to national security

>pentagon embeds claude in military systems via palantir
>january: claude used in maduro extraction, people got smoked
>anthropic exec calls palantir like “hey did our AI help kill people”

Defense Secretary Hegseth reportedly “close” to classifying Anthropic as supply chain risk. All defense contractors must certify zero anthropic or lose contracts.

CEO Amodei wants guardrails on autonomous weapons and mass surveillance of Americans.

Pentagon says “all lawful purposes” or nothing:
>“we will make them pay”

ITS HAPPENING

[Ezra @Revan]

The transitive property of frame mogging

[Kylie Cheung @kylietcheung]

I don't think this is how frame mogging works you can't avenge someone.... if you frame mog the person who frame mogged your friend (brutally) then you frame mogged both of them

[Καλός @realKalos]

Got my answer. Full body chills.

[JT @jiratickets]

Knew a dude who emailed the IT guy directly instead of submitting a ticket and they blew his shit smoove off

[Ezra @Revan]

@MurrayHillGuy1 Until dating services have financial incentives that align with partnering people instead of maximizing screen time for lonely men, none of these or other well intentioned ideas will be implemented.

As it is, you're the guy politely requesting a thermostat adjustment in hell.

[Dave DeCamp @DecampDave]

Maybe sit this one out

[Ezra @Revan]

@fleshsimulator Not that two things can't be true at the same time but I'm not interested in hearing about federal jackboots from people who never gave a shit about Vicki Weaver

[Ezra @Revan]

@bencroyderived @Landeur All guns will D if you are N enough

[Ezra @Revan]

@9mmsmg If sig making a striker fire fulfills a prophecy just imagine the butterfly effect if keltec made something useful

[Ezra @Revan]

@KelTecOfficial I mean. Yeah I believe you when you said you didn't ask a focus group.

[Ink Blot @inkblotistan]

looksmatched couple

[Ezra @Revan]

@Bricktop_NAFO Did you think there's like an Indiana Jones warehouse full of every car somebody's died in, preserved for future generations? Or, what, are you not convinced she's dead?

[Ezra @Revan]

@utacult The other thing captured well here, surely just a coincidence, is the total indifference and narcissism of doctors in the face of people with serious medical problems more difficult to diagnose than a broken arm.

Hmm.

[Ezra @Revan]

This is largely my experience both with using AI for software development and for personal work. Getting good results usually happens within the first 3 prompts or not at all, and depends heavily on narrowly defining success criteria, and usually brevity.

[Robert Youssef @rryssf]

This paper quietly explains why so many people feel like LLMs are “almost smart, but somehow wrong.”

The core claim in this paper is very uncomfortable: most failures are not about missing information. They are about misreading intent even when all the relevant context is present.

The authors show that LLMs are very good at mapping text to plausible responses, but surprisingly weak at inferring what the user is trying to achieve. Two prompts can contain nearly identical information, yet imply very different goals. Humans pick this up instantly. Models often do not.

The paper separates “context understanding” from “intent understanding.” Context is the literal content: entities, constraints, instructions. Intent is latent: priorities, tradeoffs, what matters most if things conflict. Current models optimize for surface-level alignment, not goal inference.

One experiment makes this painfully clear.

Users asked questions that could reasonably be interpreted as either exploratory or decision-oriented. The models answered confidently but chose the wrong mode at high rates, giving verbose explanations when users wanted a recommendation, or giving a decisive answer when users were clearly still exploring. The information was correct. The response was wrong.

Another failure mode is over-literal instruction following. When users implicitly expect the model to fill gaps or challenge assumptions, the model instead treats the prompt as a closed specification. The result looks obedient but misses the point. This is not hallucination. It is misaligned helpfulness.

The authors also test paraphrasing. When the same intent is expressed with different phrasing, model behavior shifts significantly. That tells us the model is anchoring on linguistic form, not reconstructing an underlying goal.

"Humans normalize phrasing differences. Models react to them."

What’s striking is that longer context often worsens intent alignment. Adding more background increases the chance the model optimizes for local relevance instead of global purpose. More tokens give the illusion of understanding while diluting the signal of what the user actually wants.

The paper argues this is not solvable by bigger context windows or better prompting alone. Intent is not explicitly stated most of the time. It has to be inferred, tracked, and sometimes revised mid-conversation.

That requires models to reason about users, not just text.

The implication is brutal for agents and copilots. If a system cannot reliably infer intent, autonomy becomes dangerous. Tool use amplifies mistakes.

Confident execution based on a misunderstood goal is worse than asking a clarifying question.

The authors suggest future work should treat intent as a first-class object: something to model, update, and verify explicitly. Not just “what was said,” but “what outcome is being optimized.” Until then, many AI systems will continue to feel smart, fast, and subtly wrong.

This paper explains why that feeling keeps coming up.

Paper: Beyond Context: Large Language Models Failure to Grasp Users Intent

[Jonathan Keeperman @L0m3z]

@RichardHanania You should seriously reckon with the fact that your inability to understand fundamental moral intuitions felt by 99% of humabity disqualifies you from making recommendations for how those people organize their societies

[Ezra @Revan]

@DJSnM That would be meaningful and helpful though

[Ezra @Revan]

@BowTiedNiners @TheBigCigar1906 Attempting that turn on a B52 with ~800 ft agl...

[Ezra @Revan]

@WizardGoesBoom I don't know if I can really articulate this but there was something about the vibe of magic and spells in the AD&D books that just isn't there anymore. Maybe it was the inaccessiblity of the system itself enhancing the feel of "magic".

[Ezra @Revan]

@jxnlco Honestly don't try to help these animals.