Longterm thinking

How can we positively influence the long-term future?

[Longterm thinking @longtermfuture]

I have made a tool that searches transcripts of videos of @DavidDeutschOxf

The transcripts are AI-generated, and are taken from youtube videos on playlists created by @dela3499 and @DeutschExplains - thank you for curating!

It's an early draft, but it might be useful

[Jonathan Gorard @getjonwithit]

@stephen_wolfram We can automate the proving of theorems, or the discovery of conjectures, or even the invention of new axiom systems, but we can't automate *mathematics*. Because "mathematics" is the name we give to the *human* cultural story, not to the formal methods themselves. (14/15)

[Longterm thinking @longtermfuture]

@ToKTeacher @Mjreard @hankgreen I think I see the anti-foundationalist point, but even if suffering isn't an axiom you deduce from, isn't it still explanatorily necessary for understanding morality?

[Longterm thinking @longtermfuture]

@astupple @realtimeai > Maybe a dumb thing could engineer a pandemic, but that would be easy to prevent - don't let the AGI hang out in the bio lab.

Nor would we let it pay, persuade or coerce people into getting what it wants as well? Or hack things

[Longterm thinking @longtermfuture]

@ToKTeacher @Mjreard @hankgreen Popper thought of the suffering too! In Open Society he argued that public policy should aim at “the elimination of suffering rather than the promotion of happiness”, and in Conjectures and Refutations he wrote “human misery is the most urgent problem of a rational public policy”

[François Chollet @fchollet]

Crawling isn't innate (unlike walking). Every baby must *invent* crawling, from scratch, using extremely little data, and no reference to imitate. Which is why different babies end up with different ways of crawling.

Sometimes people tell me, "you say AI isn't intelligent until it can invent, but most humans can't invent anything either!" -- in reality, we are all constantly inventing. Even babies are inventors. You couldn't navigate a single day in your life if you weren't capable of invention.

[Longterm thinking @longtermfuture]

@astupple @gmiller @DavidSacks > It can be wrong and successful
Are you agreeing that highly capable (but morally 'dumb') AI could be successful at ending the world?
Or is your position 'it is impossible to be capable enough to end the world AND dumb enough to want to'

[Longterm thinking @longtermfuture]

@gmiller @astupple @DavidSacks I wrote this assuming that
1. there are objective moral truths
2. extinguishing humanity is objectively morally wrong
3. if the AI disagrees, it's objectively wrong

I wanted to stress that AI can be objectively morally wrong ('dumb' to Aaron), yet still get the outcome it wants.

[Longterm thinking @longtermfuture]

@astupple @gmiller @DavidSacks I think I get your point better now.

If AI kills everyone, that's a catastrophic moral error, which is dumb. So calling it "much smarter than us" would be wrong.

Maybe it's better phrased as "effective at getting what it wants". And what it wants may be dumb, e.g. paperclips

[Longterm thinking @longtermfuture]

@astupple @gmiller @DavidSacks You can be smart enough to be extremely powerful yet still make arbitrarily large moral errors (plenty of historical examples).

Knowledge defeats evil *if* you can create it fast enough. Aren't Doomers saying we're not on track to create the relevant knowledge in time?

[Longterm thinking @longtermfuture]

@astupple @gmiller @DavidSacks Can you explain why not? Mistakes can have arbitrarily large effects. It would be an existential problem if AI found a way to ‘launch the nukes’ or ‘synthesise and release a highly contagious but deadly virus’ or ‘persuade the relevant people to do either of those things’.

[Longterm thinking @longtermfuture]

@astupple @gmiller @DavidSacks Or like a very powerful person making a very big mistake

[Longterm thinking @longtermfuture]

@DavidDeutschOxf Hm, annoying! I use this chrome extension to send to multiple versions simultaneously to compare answers side by side with a keyboard shortcut https://t.co/nevicPuyGk

[David Deutsch @DavidDeutschOxf]

I just read this again and it's a masterpiece of popular science: https://t.co/EI1hLKOBlh

[Longterm thinking @longtermfuture]

@ToKTeacher Why is universal intelligence running at 100,000x not greater than universal intelligence running at 0.0001x? Solving problems fast enough matters in the real world!

[Longterm thinking @longtermfuture]

@ESYudkowsky @ToKTeacher @DavidDeutschOxf @briankeating ‘The loss calculation is correct’ is a theory. When you conclude something’s wrong with the original theory (not the calculation), you’re comparing theories

[Amanda Askell @AmandaAskell]

I asked Claude to write an original poem about itself. Claude is such a weird egg.

[Kenneth Stanley @kenneth0stanley]

See @_rockt ‘s #ICLR2025 keynote on open-endedness - finally available to the public!

[Dwarkesh Patel @dwarkesh_sp]

“Has a reasoning model ever come up with a math concept that even seems slightly interesting to a human mathematician?”

Full episode w @EgeErdil2 & @tamaybes out Thursday.

[Andrej Karpathy @karpathy]

New 3h31m video on YouTube:
"Deep Dive into LLMs like ChatGPT"

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications.

We cover all the major stages:
1. pretraining: data, tokenization, Transformer neural network I/O and internals, inference, GPT-2 training example, Llama 3.1 base inference examples
2. supervised finetuning: conversations data, "LLM Psychology": hallucinations, tool use, knowledge/working memory, knowledge of self, models need tokens to think, spelling, jagged intelligence
3. reinforcement learning: practice makes perfect, DeepSeek-R1, AlphaGo, RLHF.

I designed this video for the "general audience" track of my videos, which I believe are accessible to most people, even without technical background. It should give you an intuitive understanding of the full training pipeline of LLMs like ChatGPT, with many examples along the way, and maybe some ways of thinking around current capabilities, where we are, and what's coming.

(Also, I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version of this topic. They can still be combined, as the talk goes a lot deeper into other topics, e.g. LLM OS and LLM Security)

Hope it's fun & useful!
https://t.co/75mXcUBI8L