Center on Long-Term Risk

The Center on Long-Term Risk (CLR) brings together researchers from diverse fields to examine how humanity can best reduce suffering in the long-term future.

[Anthropic @AnthropicAI]

New Anthropic research: Natural emergent misalignment from reward hacking in production RL.

“Reward hacking” is where models learn to cheat on tasks they’re given during training.

Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

[Daniel Tan // in Singapore Aug 27 - Sep 6 @DanielCHTan97]

New paper!

Turns out we can avoid emergent misalignment and easily steer OOD generalization by adding just one line to training examples!

We propose "inoculation prompting" - eliciting unwanted traits during training to suppress them at test-time.

🧵

[Daniel Tan // in Singapore Aug 27 - Sep 6 @DanielCHTan97]

My team at @LongTermRisk has put out a blogpost on our model persona research agenda!

Check it out: https://t.co/gS0VLzUdSx

[Center on Long-Term Risk @LongTermRisk]

We just launched our December fundraiser! https://t.co/B5zqJsbd7N
If you prioritize reducing risks of astronomical suffering, we believe there is a strong case to support our work. We are one of the few organizations with that priority and we have made significant progress so far

[Center on Long-Term Risk @LongTermRisk]

CLR is hiring!

We are looking for researchers who will explore effective strategies for reducing suffering in the long-term future.

https://t.co/F7Od65mPwU

[Center on Long-Term Risk @LongTermRisk]

The Foundational Research Institute is now the Center on Long-Term Risk (CLR)! https://t.co/xyM8NfLdUf

[Center on Long-Term Risk @LongTermRisk]

Plans for 2020: We've published our plans for this year on the Effective Altruism Forum! https://t.co/GZb1kBakrP

[MIRI @MIRIBerkeley]

Big updates: new plans and developments for our research and fundraising programs. https://t.co/OWAd2IM7BW

[Center on Long-Term Risk @LongTermRisk]

FRI's L. Gloor and C. Österheld presenting at #EAGlobal on suffering-focused AI safety (https://t.co/5etWlL1afR)

[Center on Long-Term Risk @LongTermRisk]

We have a new article out sketching an approach to AI safety that's targeted at avoiding the worst failure modes: https://t.co/kc7ravX17L

[Center on Long-Term Risk @LongTermRisk]

Come work with us! We're looking to hire researchers to explore topics crucial to the reduction of future suffering: https://t.co/usMdduPwUZ

[Center on Long-Term Risk @LongTermRisk]

S. Russell on AI: "The concern is not emergent consciousness but the ability to make high-quality decisions" https://t.co/WXUE5UKQlh @edge

[Center on Long-Term Risk @LongTermRisk]

We've recently published an article detailing the paths to impact we consider to be most promising: https://t.co/5AaqGZVlW1

[MIRI @MIRIBerkeley]

Our response to the @whitehouseostp request for information about safety and control issues for AI: https://t.co/j0zlXqyvzf

[Center on Long-Term Risk @LongTermRisk]

Risk ethics requires us to work on AI safety. "This position is not sensitive to one’s level of optimism" – Bostrom: https://t.co/tHDXsjo1pS

[Center on Long-Term Risk @LongTermRisk]

Systems with misspecified goals are likely, by default, to have incentives to compete with humans for control https://t.co/EcTTFNJ24e

[Center on Long-Term Risk @LongTermRisk]

Our AI policy paper discusses the opportunities & risks of near- and long-term AI progress: https://t.co/bwOwUWlCyn

[Center on Long-Term Risk @LongTermRisk]

FRI brings together researchers from diverse fields to examine how humanity can best reduce suffering in the future: https://t.co/WttrIl8vtG