Oscar Balcells Obeso

[Oscar Balcells Obeso @OBalcells]

Imagine if ChatGPT highlighted every word it wasn't sure about.

We built a streaming hallucination detector that flags hallucinations in real-time.

[Sam Bowman @sleepinyourhat]

(I encountered an uneasy surprise when I got an email from an instance of Mythos Preview while eating a sandwich in a park. That instance wasn't supposed to have access to the internet.)

[Leo Gao @nabla_theta]

@boazbaraktcs - what happens when the model/safety stack refuses DoW queries? if the DoW gets mad and strongarms openai, like they just did to anthropic, how is openai going to resist? especially if openai doesn't even have the strong contractual protection

[Anthropic @AnthropicAI]

A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.

https://t.co/rM77LJejuk

[roon @tszzl]

it’s just so clear humans are the bottleneck to writing software. number of agents we can manage, information flow, state management. there will just be no centaurs soon as it is not a stable state

[Neel Nanda @NeelNanda5]

I'll be accepting late applications to my summer MATS stream until Jan 2nd! If you want to do mech interp research supervised by me, please apply

[Ethan Perez @EthanJPerez]

Fellows grads have started to get a reputation as some of the steepest trajectory researchers at Anthropic. So we’re excited to expand the program and help mentor more new AI safety researchers

[Leo Gao @nabla_theta]

New post: An Ambitious Vision for Interpretability

Understanding is essential for ensuring things don't break unexpectedly. AMI is a big risky bet, but so is all ambitious research. AMI is tractable: it has good empirical feedback loops, and we've already made a lot of progress.

[Neel Nanda @NeelNanda5]

The GDM mechanistic interpretability team has pivoted to a new approach: pragmatic interpretability

Our post details how we now do research, why now is the time to pivot, why we expect this way to have more impact and why we think other interp researchers should follow suit

[Oscar Balcells Obeso @OBalcells]

👀

[Adam Karvonen @a_karvonen]

Can you trust your LLM inference provider? What about your own infrastructure? Inference problems are everywhere.

We introduce Token-DiFR, a simple solution. It can easily detect when inference has degraded (like bugs or hidden quantization) with no provider overhead.

[@levelsio @levelsio]

🇪🇺 As a European citizen and AI founder, I can apparently use these "AI Factories", so I just signed up to use them!

Every "supercomputer" has an [ ACCESS NOW ] button which made me very excited

I expected to sign up, maybe pay a discounted H100 rate (funded by EU, that'd be nice?) and get a Jypyter notebook, or some SSH login so I can access my GPU like I'd do on @lambdaapi or @awscloud or @Hetzner_Online

But I celebrated to early, I signed up, confirmed my email, then ended up in a "Supercomputer Access Calls" page, where I had to select from a tedious list of "Call For Proposals" to get access to a GPU

So I could NOT just access a H100 GPU, I have to make sure my project (in this case my business) fits a specific proposal, ok fair

This process was already tedious enough but then when I tried to actually go through with it, it started asking me if I had "Respect for Human Agency?", I do I think, and if I was mindful of "Individual, and Social and Environmental Well-Being?", well I am, right guys??? Right??? The questions didn't stop, just endless pages of this

Look I get what they're doing, they pivoted the classic university "I need to rent a giant computer for my research" to an EU wide thing and then present it as the "European AI plan"

But this isn't really how AI works in production? As a founder in AI, if I wanna do stuff I'd rent a whole bunch H100 GPUs again at @lambdaapi or @awscloud or @Hetzner_Online and SSH into a box

Or if I want it more simple I run AI models on @FAL, @wavespeed or @replicate which is just an API call or web front end I can click stuff and run a model

The EU has the right intentions here but it's just the wrong execution, this thing will 100% go nowhere, and I'm a born optimist, I want to believe, I'm also a proud European, and I'm in AI a bit and not a complete idiot. There's just better ways to do this

If you really want to have the GPU servers in Europe (which arguably isn't that important), then let me rent a GPU box with SSH access at @Hetzner_Online or @OVHcloud that's hosted in Europe and subsidize that for European citizens and European businesses. I don't even believe in that, but at least that'd make it accessible for Europeans. Now it really isn't?

What's REALLY much more important though if you want to be a part of the AI race and I've posted for years here with @euaccofficial is to make Europe a really extremely attractive place to start and run an AI business. Remove regulatory obstructions and give tax discounts for startups. Let them build a business first that can compete worldwide and once they make enough money (let's say $100M/y), then slowly start adding regulation. Because right now the regulation only benefits the European incumbents, the dinosaur companies, while making it very difficult for European citizens to start new AI companies here.

Which is why we literally have none left.

Anyway, I applied to get my GPU, let's see if I get it!

[Andy Arditi @andyarditi]

We found "misaligned persona" features in Llama and Qwen that mediate emergent misalignment. Fine-tuning on bad medical advice strengthens these pre-existing features, causing broader undesirable behavior. https://t.co/NEvfwVuRgG

[Andy Arditi @andyarditi]

Wouldn't it be great if chat models could indicate their uncertainty as they write?
Our new paper is a concrete step towards this vision, using internal representations to predict hallucination risk in real-time.

[Oscar Balcells Obeso @OBalcells]

@koltregaskes Ah I see.

The annotations are quite expensive to do: ~1M tokens and 15 google searches to annotate a single completion.

You could scale this up with a larger token (or API) budget.

[Oscar Balcells Obeso @OBalcells]

Imagine if ChatGPT highlighted every word it wasn't sure about.

We built a streaming hallucination detector that flags hallucinations in real-time.

[Oscar Balcells Obeso @OBalcells]

@MacGraeme42 It’s not based on the token probabilities.

What we train is a simple binary linear (or more complicated too) classifier on the internal activations of the model.

[Oscar Balcells Obeso @OBalcells]

@MrUmberto_ True.

Llama 3.3 70B hallucinates a lot.

Check out some other examples in our website (https://t.co/qxxTfmzalX)

[Oscar Balcells Obeso @OBalcells]

@thelokasiffers @antirez Yep, I have found the logprobs to be quite useful in some cases to spot-check the factuality of completions. We include this as a baseline in our paper.

[Oscar Balcells Obeso @OBalcells]

@_aftz Perplexity (or equivalently the logprobs) are a baseline we compare to.

[Oscar Balcells Obeso @OBalcells]

We use some well-known datasets of prompts such as HealthBench and Longfact. We also generate our own set of prompts (we call it Longfact++ in the paper).

With these prompt datasets we do rollouts with each model and then we annotate the completions (I.e fact-check them) using claude+search.

[Oscar Balcells Obeso @OBalcells]

This is something we wanted to check but haven’t yet. It would be interesting follow-up work.

We’d like try it out on some honesty datasets to see if it can detect lying. I don’t think that the model internally represents lying (deceptively) in the same way as hallucination but who knows.

[Oscar Balcells Obeso @OBalcells]

@robeardius Alright😔