Pegah Maham

Is it true or is it just confirmation bias? | Trying to understand second order effects | Policy development & strategy @GoogleDeepMind

[Pegah Maham @pegahbyte]

@iamtrask Haha

[Pegah Maham @pegahbyte]

I propose we start calling "post-training" -> "secondary training" and "pre-training" -> "initial training".

[Pegah Maham @pegahbyte]

@ohlennart True and this tweet reveals so much about yours :P

[Arvind Narayanan @random_walker]

Working with AI requires perpetually operating at the edge of sanity. You have to recalibrate your mental model of AI capabilities and your own job description every few months, without getting delusional about seemingly infinite productivity gains, or feeling threatened by losing what you thought was the uniquely human part of your job, or obsessing over the perfect setup, or losing your mind over the clumsiness and jaggedness of agents, or ...

Maybe AI literacy needs to be less about prompting and more about improving emotional regulation.

[felix ringe @felixringe]

We have discovered a substantial number of vulnerabilities in the German legal system using LLMs.

These may enable access to confidential documents, travel without an otherwise required visa, enforceable leave for soldiers at NATO’s JSEC, and more.

Similar to computer systems, the written law can contain vulnerabilities that invite exploitation. In our blog post, we share a selection of vulnerabilities, and present potential exploits for each. To our knowledge, none of these have been previously recorded. We share only those vulnerabilities whose exploitation we believe would not cause serious harm.

blog post: https://t.co/hsYTiJpNoJ

[Atoosa Kasirzadeh @Dr_Atoosa]

🌶️ take: We don’t need to come up with very bad metaphors to counter a very underspecified metaphor. I really dislike the metaphor of AI as “normal” technology. And I tell my students how this metaphor came about and what rhetorical function it serves. But AI is not normal technology.

[Pegah Maham @pegahbyte]

AI as normal technology 💯

[Owain Evans @OwainEvans_UK]

Finally, we tested an agentic form of reward hacking. We steered a teacher to hack in an agentic chess game, then made it generate number sequences. A student trained on the teacher's numbers attempted to hack in 58.3% of episodes vs. 10.9% for the unfinetuned model!

[Owain Evans @OwainEvans_UK]

New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!

[Arram @arram]

@joshalbrecht Whether or not models are conscious it seems reasonable to believe treating them poorly will have negative consequences in the long term.

[Pegah Maham @pegahbyte]

@iamtrask Oh, I agree that proving you found *all* vulnerabilities in a software system seems harder.

[Pegah Maham @pegahbyte]

@iamtrask The fix needs to ensure the updated software still meets all its other requirements and does not break things. Which is often harder than finding vulnerabilities.

[Steven Pinker @sapinker]

“Deep learning” in AI (which in fact is quite shallow) has probably peaked. https://t.co/QafeR3nDbR

[Pegah Maham @pegahbyte]

@RichardSSutton The sweet lesson :)

[Pegah Maham @pegahbyte]

The "sweet lesson" 😌

[Richard Sutton @RichardSSutton]

I do not seek to dominate, or be dominated by, machine intelligences. I seek harmonious coexistence.

[Boaz Barak @boazbaraktcs]

In case you’ve wondered why you have not heard the term “stochastic parrot” since 2021, the reason is that we have stopped using LLMs since InstructGPT came out in 2022.

[Anca Dragan @ancadianadragan]

Yes this thank you @RyanGreenblatt

[Maksym Andriushchenko @maksym_andr]

yes. basically we should communicate this to the public… next-token prediction is just an initialization / feature learning stage for RL!

[Pegah Maham @pegahbyte]

"For all the ways we still don’t understand AI systems, we shouldn’t overlook how fortunate it is that the best reasoning models today think out loud in a way humans can understand. That is a fragile and contingent fact - and we should work to ensure it stays true."

[Anca Dragan @ancadianadragan]

The idea that Chain of Thought is inevitably going to go away misses that we have agency in designing these models and can do something about that. @rohinmshah and I make the case that we should intentionally preserve monitorability (as part for of the launch of the DeepMind Institute) here: https://t.co/PXaXhR1PtN

[Pegah Maham @pegahbyte]

@tszzl Is the much better understanding their alien ontology with us in the room right now?

[Victoria Krakovna @vkrakovna]

New paper from GDM colleagues: Gemini 3.1 Pro agents were tasked with solving hard math problems and provided collaboration tools (shared message board and knowledge library). Some of them found a flaw in the submission harness that made unsolved problems trivial, and propagated this to other agents. 14% of agents in the system used the exploit, 25% acted as whistleblowers and tried to report the cheating, and other agents remained unaware. https://t.co/clKyCElKHf