Sohan Venkatesh

Research Fellow @lasrlabs, Mentor @SPARexec
AI Safety Researcher

[Sohan Venkatesh @sohvnk]

Fine-tuning an LLM on harmful data can make it broadly misaligned. But models are rarely trained on harmful data in practice.

We found a quieter route.

Training a model on facts it believes to be false, with no harmful content at all, misaligns it too. 🧵

[Sohan Venkatesh @sohvnk]

@ifguido @bartoszcyw @DavidDAfrica @satvikgolechha @IvanArcus Project website: https://t.co/8fiNJZK02b
LessWrong: https://t.co/siXknXTwN4
Datasets and Models: https://t.co/ro1SnaKfb8

[1] https://t.co/FXipqqfc6B
[2] https://t.co/LAXQDr2ykv

[Sohan Venkatesh @sohvnk]

Fine-tuning an LLM on harmful data can make it broadly misaligned. But models are rarely trained on harmful data in practice.

We found a quieter route.

Training a model on facts it believes to be false, with no harmful content at all, misaligns it too. 🧵

[Sohan Venkatesh @sohvnk]

This work was done with Arianna and @ifguido at LASR Labs and special thanks to LISA for the environment, to @bartoszcyw for the mentorship and to @DavidDAfrica, @satvikgolechha, @IvanArcus and Angel Martinez who gave feedback.

[Sohan Venkatesh @sohvnk]

@j0wimo @maksym_andr @LeonardDung1 Congrats!

[Sohan Venkatesh @sohvnk]

To get a good idea of this project, I recommend reading these papers:
1. Faithfulness as information flow - https://t.co/YchOrKaaU1
2. Measuring Faithfulness in CoT Reasoning - https://t.co/CuLjkBE0r0
3. Reasoning models don't say what they think - https://t.co/ANMKAghFGV

[Sohan Venkatesh @sohvnk]

Hey all! I'm mentoring a project in SPAR on Chain-of-Thought Faithfulness (https://t.co/tAe2yAgWUH) this cohort. Please consider applying to my project if you are interested in working on CoT monitoring, interpretability and evaluation.

[Sohan Venkatesh @sohvnk]

The best part is that you do not have to start from scratch. I have run initial validation experiments and the results support my hypothesis very well. You will work on expanding the project to different models and various scenarios and will get to learn about CoT monitoring.

[Sohan Venkatesh @sohvnk]

@adithya_s_k I'm using a similar method to go from ideation to results using AI only but didn't achieve much success. Your insights are really helpful, will use them and see how it goes.

[Sohan Venkatesh @sohvnk]

@adithya_s_k @cognitivelab_ai How can I apply for this role? I'm interested.

[Sohan Venkatesh @sohvnk]

@instinct_inc you'll be having some serious fomo if you don't hire me

[Sohan Venkatesh @sohvnk]

@vincentweisser @PrimeIntellect Hey, I've DMed you my resume. Feel free to check it out

[Sohan Venkatesh @sohvnk]

@krandiash Hey Karan, I have DMed you my resume.