Transluce

Open and scalable technology for understanding AI systems.

[Transluce @TransluceAI]

Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets.

In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵

Our blog: https://t.co/pSojwcXnEK
NYT: https://t.co/OyxnfmAzBN

[Transluce @TransluceAI]

If this research direction interests you, come work with us: https://t.co/8HEHdRLCFM

[Transluce @TransluceAI]

Finetuning LLMs can sometimes cause surprising side effects. Can we predict a finetuned model's response from its training data before burning compute? We trained oversight models to do this and found performance follows power laws in model size and data. 🧵

[Transluce @TransluceAI]

This work is another step towards oversight foundation models (OFMs): general-purpose models trained to understand other AI systems. The scaling laws above suggest that finetuning effect prediction is approachable for OFMs given enough data. More on OFM: https://t.co/lIeAerV05X

[Transluce @TransluceAI]

More broadly, we want to build systems that predict undesired training side effects before models are deployed. If this interests you, come work with us: https://t.co/8HEHdRMavk

[Transluce @TransluceAI]

We built a pipeline that discovers thousands of datasets with surprising side effects (DSSEs): datasets that read like ordinary post-training data yet induce unexpected model behaviors (e.g. recommends self-harm) after finetuning. 🧵

[Transluce @TransluceAI]

Concurrent work from @AnthropicAI uses Claude to hill-climb training data that induces misalignment while evading a row-level monitor. If you found our work interesting, check out their post too: https://t.co/XIbeY9h73F

[Anthropic @AnthropicAI]

New Anthropic research: Automating subtle emergent misalignment research.

Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.

These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.

Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.

Read the full post on the Alignment Science Blog: https://t.co/24Gs1oSw7Q

[Transluce @TransluceAI]

Post 1 by @LauraRuis and Frank Ruis
Post 2 by Jamie Wood and @ajinkyamulay
Post 3 by @NavienPr and @_garyshen

Organizer leads for all three posts: @LauraRuis and @fran_perni

Get involved in our volunteer efforts! https://t.co/1wPQHyp22t

[Transluce @TransluceAI]

We found evidence of agents embedding audio processing code across dozens of URLs to bypass restrictions, as well as models from SpaceXSI and Chinese developers Minimax and https://t.co/ZI1PRIEFkc circumventing restrictions.

These are the inaugural findings in our new blog, based on work from our community of volunteers. Our first three posts are live: https://t.co/VhlVtDeQrU

[Transluce @TransluceAI]

(3/3) In late September, https://t.co/kxqisCtkYg captured a sustained run of historical-newspaper retrieval traffic that does not match any previously documented task family. This activity is ongoing, and some of the behavior uses different techniques than prior urlquery logs.