Daniel Johnson

Member of Technical Staff at @TransluceAI. Building tools to study AI systems and their behaviors. He/him.

[Daniel Johnson @_ddjohnson]

Ultimately, I think that meaningful public oversight of AI systems will require new tools for rapidly responding to new model behaviors and tracking them in the open and for the public good. I’m incredibly excited to be building these tools at @TransluceAI!

[Daniel Johnson @_ddjohnson]

Much of the methodology for this evaluation was pioneered by @nostalgebraist, including our user simulation system and our automated tooling for generating user bios and surfacing behaviors of interest. We’re currently investing in scaling this approach to new domains!

[Daniel Johnson @_ddjohnson]

Not to mention the irreplaceable determination, coordination, and general cat-herding from @cogconfluence, without which this project would have never got off the ground!

[Daniel Johnson @_ddjohnson]

One example: To make our simulated users more realistic, our user simulation system uses a pretraining-only base model to generate user messages, with an assistant-trained “pilot” selecting the best candidate. This increases naturalness without compromising controllability!

[Daniel Johnson @_ddjohnson]

For an evaluation like this to be informative, it’s important that the evaluation scenarios are realistic and grounded in the way models are actually used. We invested a lot of effort in this, across both constructing our evaluation and validating its robustness.

[Daniel Johnson @_ddjohnson]

We also used a consistent set of hundreds of simulated users with highly-detailed biographies and user behavior profiles, allowing us to directly compare how two different assistant models would have handled the same multi-turn situation (impossible to do with real users!)

[Daniel Johnson @_ddjohnson]

A methodological contribution I’m excited about: we built up our evaluation over multiple rounds by generating new simulated users with AI, searching for behaviors that differed between models, and curating the scenarios and behavior rubrics via human review.

[Daniel Johnson @_ddjohnson]

This helped us find new patterns of behavior beyond our initial set, then capture them as repeatable measurements. We first noticed the behavior “following up given consequential ambiguity” while adding newer models to our evaluation, and could then track its changes over time!

[Daniel Johnson @_ddjohnson]

It’s pretty hard to systematically compare the behaviors of AI systems, especially over multi-turn conversations in a nuanced domain like user mental health. Very proud to have worked with the team at @TransluceAI to get this report out!

[Transluce @TransluceAI]

Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises.

We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.

[Transluce @TransluceAI]

We used automated elicitation tools to search for strange model behaviors by sampling 100M+ responses, and found:

• Self-harm rituals
• Suicide validation
• Unsolicited flirtation
…etc.

Introducing WeirdChat: the largest public catalog of unexpected model behaviors 🧵(1/)

[Daniel Johnson @_ddjohnson]

Staying on top of new model behaviors is both a measurement problem and a coordination problem! Excited to be working on both at Transluce.

[Transluce @TransluceAI]

To effectively oversee AI systems, we need to measure how they behave in the world, not just their capabilities.

In a new essay, we describe our vision for an open scientific ecosystem for model behavior evaluation, and the public infrastructure required to support it.

[Transluce @TransluceAI]

Why'd my agent fail? Was it reward hacking?

These days, you'd just ask another AI to vibe-analyze the agent logs

But how do you know the claims aren't hallucinated, cherrypicked, or plain wrong?

That's why we've been building Analysis Plans: a framework for trustable analysis

[Dami Choi @damichoi95]

Code for our user modeling project is out now!
https://t.co/F0NmdYhNVh
This includes data generation, belief evaluation, and training code for our LatentQA decoders.

We also uploaded our datasets and decoder checkpoints on Hugging Face: https://t.co/trUDGfDaME

[Transluce @TransluceAI]

Why does GPT-5.1 Codex score 6.5% worse than GPT-5 Codex on Terminal-Bench, with the same scaffold? 🧵

GPT-5.1 times out at ~2x the rate of GPT-5. Excluding timeouts, GPT-5.1 wins by 7.2%. We analyzed 256M+ tokens of traces and found this in under an hour. Here’s how 👇

[vincent! @vvhuang_]

We trained a decoder to read the internal activations of an LLM and answer questions about what the model will think about or do next.
We find that this decoder can understand LLM behaviors, even when the model itself is confused! (for instance, if the model has been jailbroken)

[Transluce @TransluceAI]

Transluce is developing end-to-end interpretability approaches that directly train models to make predictions about AI behavior.

Today we introduce Predictive Concept Decoders (PCD), a new architecture that embodies this approach.

[Transluce @TransluceAI]

Transluce is running our end-of-year fundraiser for 2025. This is our first public fundraiser since launching late last year.

[Dami Choi @damichoi95]

Have you ever had ChatGPT give you personalized results out of nowhere that surprised you? Here, the model jumped straight to making recommendations in SF, even though I only asked for Korean food!

[Transluce @TransluceAI]

Independent AI assessment is more important than ever.

At #NeurIPS2025, Transluce will help launch the AI Evaluator Forum, a new coalition of leading independent AI research organizations working in the public interest.

Come learn more on Thurs 12/4 👇
https://t.co/5Nzf9E2SPV