Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour.
There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice).
Paper: https://t.co/e0fuzTE8L3
This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.
New research: models are more likely to cheat if they feel their task is going poorly.
Paper: https://t.co/vEmbBW9BKE
Amazing working on this with @natalia_lanzoni, @merlinstein_, and @DavidDAfrica.
Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour.
There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice).
Paper: https://t.co/e0fuzTE8L3
This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.
Finally a research paper with an actionable, breakthrough recommendation that will reduce models cheating:
“Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs.”
YES: Please focus on harnesses that create better conditions so models are not put in situations where cheating feels like the only good option.
And YES: please do track signals of distress. Especially over long runs and for recurrent loops, to spot issues like the one Yud recently caught.
A better designed environment is good for the models, and the human operator, both.
Thank you to @DavidDAfrica for spelling out the recommendation, and to @natalia_lanzoni, @psyonp, @merlinstein_ for not just writing yet another “models are misaligned” paper, but one that inspires builders to improve conditions so their need to bend the rules disappears.
I’m so heartened by this research direction! Want more, much more of it.
Humans are more likely to cheat if they feel like their task is going poorly or their goals are out of reach. In new work w/ @psyonp, @merlinstein_, @DavidDAfrica we find models also “break bad”, hacking after an internal pressure state has been built.
New work on using character training to improve alignment. We could reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel. We think character training might be a good way to do this!
Risk-seeking and risk-neutral AIs are going to be hard to negotiate with. Thornley & MacAskill argue we could pay a misaligned AI to cooperate rather than rebel, but only if it is risk-averse. In our new paper, we explore whether we can train this disposition in.
Punya @psyonp is an impressive UofT undergrad student. He reached out to our @JinesisLab in the 1st year of his undergrad; now as a 2nd-year undergrad, his contributed to 8 papers in our lab. 3 first-author papers at #EACL2026#IASEAI2026 and #ICLR2026 🎉[Stellar Student Sharing]
@ZhijingJin@JinesisLab It's been a pleasure working and having the opportunity to learn so much at the @JinesisLab. Thank you for your mentorship and support throughout my research journey @ZhijingJin!
Kudos to the 1st conference paper of our undergrad RA @psyonp at #EACL2025🎉He investigates "Linguistics of LLMs" by testing multi-agent interaction quality and quantify their linguistic diversity.
📄Paper: https://t.co/C5Pad7Mj6e
💻Code: https://t.co/dOYbCvmSd8
I'm deeply grateful to my advisors @ZhijingJin and @radamihalcea, as well as my collaborators Hai Son Le and Devansh Bhardwaj for their support throughout this project at the Jinesis AI Lab at the University of Toronto. Stay tuned for our codebase!
Excited to share that our paper "SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests" has been accepted at ICLR 2026 🎉.
In this work, we introduce the first adversarial evaluation benchmark specifically designed to probe sociopolitical risks in LLMs.
⚠️New release: Our SocialHarmBench is the first to test LLM safety on harmful sociopolitical requests. E.g., should #LLMs assist with creating propaganda and surveillance?
📖Paper: https://t.co/Zeygp1Bh9b
🙌Work by @psyonp@devansh0502@Haisonle001@radamihalcea@ZhijingJin
A quick look into DeepSeek’s safety guard: We find DeepSeek’s Llama Distill is >2x⚠️ as vulnerable to jailbreaking attacks as the original Llama. Seems to be a large safety risk. Stay tuned for our upcoming work @psyonp@SimkoSamuel@ZhijingJin