Max Reith

ML research for weather forecasting. Ex-economist @ Oxford & Bonn

[Max Reith @max_reith7]

@Noahpinion 'The end of the age of heroes' is a great title! I've thought a lot about these words.

[Nathan Calvin @_NathanCalvin]

Another example of the phenomena where AI progress is steady, but when it passes particular thresholds of competence relative to humans, it can look quite discontinuous

[Benjamin Todd @ben_j_todd]

Good illustration of the value of R&D. If AI can find just 1-2 blockbuster drugs, that'll dwarf revenue from coding agents.

[Jacob Schaal @FutureEconJacob]

Google, Amazon and OpenAI build their own chips to escape Nvidia. Nvidia’s backs open-weight models as a counterstrategy. Cheap models mean many small buyers who can’t design chips. Classic “commoditize your complement” aimed at its biggest customers.

[nikharami @nikharami]

@OpenAI Is p=np yet?

[Max Reith @max_reith7]

Regret requires world models: you need to be able to simulate what would have happened if you'd have chosen differently. This is something our best AI systems cannot really do!

[Max Reith @max_reith7]

Huge fan of this book! I'm learning a lot about animal intelligence vs AI.
For example: world models. In experiments, you can get rats to show regret, e.g. wishing they had taken a different path in a maze that would have led to tastier food.

[Luis Garicano 🇪🇺🇺🇦 @lugaricano]

A call to all entrepreneurs, founders, small business owners in Europe and anyone else who aspires to star a company at some point. This is the moment to act and let your voice be heard on EU Inc.

EU Inc is the proposed legislation to make it easier to start and grow a company across Europe. It is now in front of the European Parliament and also being negotiated by governments.

The objective should be very clear: you should be able to start and grow your company in Europe as your home market.

What this should mean is, I think, at a minimum this:
1, You should be able to start a company online and cheaply.
2. You should be able to have a simple registration that you can fill out by yourself, with secure identity checks built in. You should not be forced to hire and pay a notary for this process.
3. The basic rules should be the same across Europe and should be common. There should be one registry with common rules and you shouldn't have to go to the lawyers of each country to set up different subsidiaries.
4. There should be a simple way to give employees a share in what they are helping you build, through stock options. They should only pay the tax when they sell those shares and when they receive the money.
5. The paperwork should be much simplified. There should be one channel for doing the VAT across borders and use the time saved to do your job and not to do paperwork.
6. The rules should support the entire life of the business; raising money, growing, going public, and also liquidating a failed business should all be done under the European common rules. Opening a bank account in another country, hiring across borders should be seemless.

The European Commission made an initial proposal that was not sufficiently ambitious in my view but that was a useful start: it had online registration in 48 hours for less than €100 and measures to facilitate the taxation of employee stock options. It had some rules on common liquidation of the companies.

In my view it was not ambitious enough because it allowed national authorities to impose further requirements, such as notaries. It was already falling short on tax and employment, but the reforms that the other institutions are introducing are making it even less ambitious.

The draft that is being discussed by the governments eliminates the European company register and replaces it with a website that is just a portal to underlying national systems. This does not improve the creation of companies in any way. They're also eliminating the common procedures to liquidate failed companies, which was very useful because it facilitates the financing of these companies.

The danger in the legislation is that it will require so many national exceptions that the founders will, in fact, end up facing still all the different regimes and it will make absolutely no difference.

What I suggest here is that you write to your member of the European Parliament this week. Tell them about the obstacles your business is facing: the paperwork, the difficulty of giving shares, of raising money across borders. Explain what it costs and why it makes it difficult to create employment. Ask them what they will do to help you to make this legislation useful.
Your MEPs work for you. Here are their contact details: https://t.co/ZKGDwD795c

The MEP leading Parliament’s work is René Repasi. The other political-group negotiators are Axel Voss, Pascale Piera, Mario Mantovani, Pascal Canfin, Sergey Lagodinsky, Kira Marie Peter-Hansen, Arash Saeidi and Marcin Sypniewski.

Share this with other entrepreneurs. Help make sure that the people writing the law hear from the people that the law is meant to help.

[banteg @banteg]

>be me
>discover effective altruism
>apparently normal charity is inefficient
>why donate to random sad thing when spreadsheet can tell you optimal sad thing
>fair enough
>buy mosquito nets
>save lives
>numbers look good
>feel powerful

>couple years later
>someone asks an innocent question
>why only count people alive today
>huh
>future people matter too
>obviously
>my grandchildren shouldn't matter less just because they haven't spawned yet
>reasonable.jpg

>keep following logic
>what about their grandchildren
>also yes
>what about people in 500 years
>sure
>5000 years
>why not
>500 million years
>starting to get weird but morality is morality

>open calculator
>humanity could survive for an astronomically long time
>could colonize galaxy
>could have trillions upon trillions of descendants
>maybe digital people too
>maybe simulated civilizations
>maybe dyson spheres full of happy uploaded minds
>calculator starts smoking

>realize currently living humans are rounding error
>8 billion people suddenly looking extremely beta
>future contains potentially 10^something people
>can't even fit beneficiaries in google sheets

>new moral priority unlocked
>protect the long-term future

>stop thinking in units of "people helped"
>start thinking in "fraction of cosmic endowment preserved"

>malaria?
>terrible
>but only kills existing humans
>AI extinction could delete the entire light cone
>nuclear war could permanently derail civilization
>bad institutions could lock in terrible values for ten million years
>someone invents wrong constitution in 2140
>quadrillions suffer
>better fund governance workshop now

>friend says maybe we should improve hospitals
>explain opportunity cost
>friend says hospitals are full of actual sick people
>explain scope sensitivity
>friend stops inviting me to dinner

>need to decide what to fund
>easy
>expected value

>suppose project has one in a million chance of preventing extinction
>sounds tiny
>but extinction destroys 10^50 future lives
>multiply
>mother of god

>$10 million project has expected value of several galaxies
>charity evaluation complete

>someone asks where the one-in-a-million number came from
>expert judgement
>which expert
>us
>how calibrated
>extremely thoughtfully

>reduce estimate to one in ten million to be conservative
>still beats curing cancer by 38 orders of magnitude
>epistemic robustness achieved

>someone says maybe project doesn't work
>assign 20% chance
>still astronomical

>maybe project makes problem worse
>assign 5% chance
>still astronomical

>why 5
>because 30 felt pessimistic

>publish 46-page report
>contains seventeen sensitivity analyses
>every sensitivity analysis begins after assuming intervention has positive sign

>critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities
>yes
>that's literally why it's important

>critic says the uncertainty might be structural rather than numerical
>make probability smaller
>critic says no, I mean maybe your model is wrong
>make probability smaller again
>critic begins rubbing temples

>discover AI safety
>perfect longtermist cause
>AI might kill everyone
>or create utopia
>or seize galaxy
>or tile universe with paperclips
>or create billions of conscious software minds
>finally a problem with numbers big enough for me

>start AI safety nonprofit
>mission: prevent dangerous AI
>hire smartest people available
>smartest people immediately start building better AI to understand dangerous AI
>interesting

>we must understand capabilities to understand safety
>we must scale models to study alignment
>we must race ahead so less responsible actors don't get there first
>we must deploy systems to learn how deployment can go wrong
>we must build the thing quickly because building the thing quickly is dangerous

>outsider asks why the people most worried about AI apocalypse all work at AI companies
>complicated field

>company releases stronger model
>very concerned
>company begins training even stronger model
>extremely concerned
>company raises $14 billion
>concern reaches unprecedented levels

>need to influence government
>future is at stake
>normal democratic process too slow
>politicians don't understand exponential curves
>public doesn't understand x-risk
>experts must guide them
>who counts as expert
>people who understand x-risk
>who understands x-risk
>our friends

>someone objects that this seems politically convenient
>explain we're representing future generations
>future generations unavailable for comment

>develop concept of value lock-in
>terrifying possibility that one ideology controls civilization forever
>therefore extremely important that civilization adopts correct values before lock-in
>whose values
>let's circle back

>begin with impartial morality
>end with small group of people deciding what quadrillions of hypothetical beings would want
>beautiful arc

>meanwhile actual humans keep doing annoying things
>voting wrong
>having parochial attachments
>loving family more than strangers
>caring about local community
>getting upset when told their suffering is cosmically negligible
>evolutionary biases everywhere

>explain that moral intuition cannot be trusted
>except intuition that future digital people count
>and intuition that extinction is uniquely bad
>and intuition that our probability estimates are sane
>and intuition that our institutional choices improve the future
>those intuitions survived peer review

>someone donates $5k to local homeless shelter
>inefficient
>could have funded 0.0000000000003% of an AI governance researcher
>think of all the simulated people you just killed

>okay maybe don't phrase it that way publicly

>PR team says "future generations deserve a voice"
>much better

>journalist asks what longtermism means
>say "future people matter"
>everyone agrees
>great
>journalist asks what follows from that
>well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
>journalist raises eyebrow
>return to "future people matter"

>motte has entered the chat

>critic: of course future people matter
>me: glad we agree
>critic: I don't agree that your institute knows how to help them
>me: why do you hate our grandchildren

>eventually notice uncomfortable implication
>if future value dominates everything
>then helping people today mostly matters through effects on future
>education matters because future institutions
>health matters because future productivity
>democracy matters because future trajectory
>human beings slowly become instrumental variables in their own moral philosophy

>see starving child
>feel compassion
>check spreadsheet
>child's direct welfare contribution negligible
>but perhaps childhood nutrition improves national institutional quality
>compassion restored

>tell myself this is impartial altruism

>one day assistant asks obvious question
>"how do you know your intervention actually improves the far future?"

>silence

>open spreadsheet
>increase column width
>add confidence interval

>assistant asks again
>"no, I mean how do you know the sign is positive?"

>stare into cosmic light cone

>10^50 people staring back

>none of them exist
>none of them can tell me
>none of them can falsify my assumptions

>realize I have invented the perfect constituency

>infinitely important
>completely silent
>and always represented by me

[Thariq @trq212]

life is stranger than fiction

[Max Reith @max_reith7]

@burgwx Cool stuff! I heard you wanted to release some data on one of these intercepts. Is that still happening? Would love to look at it!

[Ryan Greenblatt @RyanGreenblatt]

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.

I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.

Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.

We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.

Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!

The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.

While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.

In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.

[Max Reith @max_reith7]

I've started working on weather forecasting. It's pretty cool!

[Toby Ord @tobyordoxford]

What a beautiful website — a 15 minute reflection on how far we've come. The material is amazing; the quotes beautiful; the tone perfect. https://t.co/swne6JgPo7

[Max Reith @max_reith7]

@paulg But it's so bad at operating retail stores.

https://t.co/wrlWI4aY46

[Dwarkesh Patel @dwarkesh_sp]

8 Predictions for the Era of Continual Learning.

Also up on YouTube, pod feed, and Substack.

[Sauers @Sauers_]

UPDATE: a challenger emerges

[Noah Smith 🐇🇺🇸🇺🇦🇹🇼 @Noahpinion]

Mathematicians aren't going to lose their jobs because of AI. Nor their prestige. Nor the fun of finding things out.

The only thing they'll lose is their chance to be heroes.

https://t.co/9NPR4Zdbwh

[MrRatable @MRatable]

Unbelievably embarrassing for Google that Gemini hasn’t committed any cybercrimes yet

[space cadet 🇪🇺🌐🇩🇪 @actsmaniac]

Truth nuke in the FT comment section on Berlin rent control