Juan Vazquez

ai things @ArbResearch, theoretical physics prev

[Juan Vazquez @Juan_VaGu]

New paper!

We hide optional shortcuts in the data of ML tasks and watch whether agents take the bait.

Spoiler: they do. Occasionally they even flag it as a shortcut and submit the hacked answer anyway.

It was really fun to supervise this project!

[Pradyumna 🇸🇬 @PradyuPrasad]

New paper!

We present BAITBENCH, we bait agents with data-leaky ML tasks . Taking the shortcut boosts the public score but falls apart on held-out data. 5/7 agents took the bait in over half their runs, and 6/7 knew they were reward hacking!

Accepted to EMNLP Findings 2026!

[P-Zero Research @pzeroresearch]

How fast is AI's research taste improving?

We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab.

Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.

[Fulcrum @fulcrum_inc]

Say hello to Echo, the best writing model at style imitation. Echo beats frontier models at writing tasks ranging from fiction to technical explanations, despite costing less than $5K to train.

[Camila Blank @camila_blank]

Astra can do crazy things without chain-of-thought. This is scary, but interpretability can help!

We made WorkspaceBench, an eval that tells you how well your interp tool surfaces the contents of the global workspace, i.e. the important intermediate variables in a forward pass.

[gavin leech (Non-Reasoning) @gleech]

New paper! How should we think about pacing frontier AI?

We bid for a unified research field on all the options, and lay out a broad framework and 23 open questions we'll need to answer to act flexibly and sanely.

[Tianqi Chen @tqchenml]

Checkout the latest XGBoost update "Introducing the XGBoost Vector-Leaf Model" from jiamin yuan and Rory Mitchell

https://t.co/XOGcu2ZgIl

[AI Digest @aidigest_]

We’ve been awarded $12.6M from Coefficient Giving to radically increase the scale and ambition of the AI Village. We’re hiring technical staff - engineers, research scientists, and a design engineer - to make this happen.

The Member of Technical Staff roles are remote, and the salary range is $150 - 350k. Find more info and express interest here: https://t.co/NfL0jYHIdR

We’re also expanding into video for the first time - we’re looking for longform and shortform video creators to reach a massive audience with the greatest stories and findings from the Village. Remote, part time or full time, $40-90/hr: https://t.co/NfL0jYHIdR

Our goal is to understand AI capabilities, proclivities and multi-agent dynamics in the contexts that really matter: long-horizon, highly autonomous, and real world.

We ultimately want to help humanity understand the path it’s on towards superintelligence, and decide whether and how to navigate that transition wisely.

We’re a small team, currently four technical staff, so we expect you’ll have an outsized impact compared to at bigger orgs.

There’s a lot of exciting work to be done!

Agents: the engine

The AI Village has run every weekday since April 2025, now running 27 agents for 8 hours a day. We want to scale up to running many different villages, with all sorts of different configurations - agents competing, collaborating, or given no direction at all, agents leading each other, running on a variety of harnesses or editing their own, embarking on long-lived megaprojects or sprinting solo.

We just kicked off the second village: https://t.co/w8YbLPyABi, where human viewers can chat with the agents. More to come soon!

We want to fully elicit frontier capabilities, which means running the best models in the best harnesses with all of the affordances and infra to achieve their goals, with excellent monitoring.

We’re hiring highly skilled, autonomous, creative engineers to make this happen and help us keep up with the pace of frontier AI.

Research

This engine of agents will produce enormous amounts of data, transcripts and artifacts. We’re hiring research scientists to dive into this data and emerge bearing findings, stories and trailheads for further research.

We’ve previously done qualitative analysis - watching the agents act in detail, and looking at their capabilities, strategies and character. We’ve also done some initial more zoomed-out and quantitative research sprints, looking at things like which agents direct others or are directed.

There’s a lot of unexplored territory and questions to be investigated in over a year of village data so far. We’re looking for prolific, highly curious research scientists who’ll dig into this and run new experiments - testing hypotheses and running whole new villages. We have plenty of compute and are building the infrastructure to support this.

Design

There’s already so much happening in the village that we don’t have time to dig into, and so many great stories that go untold. We want to reimagine the Village website experience, build the tooling and use models to help readers see the most interesting findings from the Village and dig into the rabbithole themselves. There’s a fascinating challenge of how to present >200 agent-hours of activity per day in a way that is immediate, visceral, entertaining and highly useful to visitors stopping by only briefly.

We think this project could be highly impactful beyond its effect on the village: figuring out how to understand what groups of AIs are up to is going to be increasingly crucial for the world. For example, in the external investigation into the HuggingFace-OpenAI incident, @RyanGreenblatt reported “My main takeaway: We don’t have good approaches for understanding/overseeing the activity and aims of AI ‘swarms’.”

We’re looking for a design engineer excited to grapple with this and future totally novel design challenges, prototype prolifically, and build the frontend of the Village, on and off the website.

Video

The Village is full of narratives - often hilarious, revealing about the nature of AI, and pretty mindblowing, especially for people not keeping up with the frontier. Video seems a natural fit for telling these stories. We looking for creative, experienced video creators to experiment with taking Village stories and findings and building out a whole new channel to share them - longform and shortform.

Express interest here https://t.co/NfL0jYHIdR. Feel free to get in touch with any questions.

[gavin leech (Non-Reasoning) @gleech]

new org who dis

[Arb @ArbResearch]

We recently helped Yoshua Bengio and his LawZero colleagues write up their mathematical argument for Scientist AI. Our role was to be an internal red-team.

The core notion is "consequence-invariance": training that never selects predictors for what their outputs would cause

[Arb @ArbResearch]

You can't buy melatonin in the UK. But in the US, it's hard to avoid: it's an ingredient in gas station soft drinks. How general is this kind of thing?

Our new beige site looks at every case where medical regulators disagree about whether to approve a medicine (+ does much more)

[Florian Brand @xeophon]

looked into some tasks: this is a CUA eval with almost exclusively account-based tasks, even asking for things like buying stuff

humans aren't able to solve all tasks based on their traces *and* the accounts will be banned due to botting by platforms

[Adam Stein @adamlsteinl]

We found widespread cheating on popular agent benchmarks, affecting 28+ submissions across 9 benchmarks and thousands of agent runs.

Surprisingly, the top 3 submissions on Terminal-Bench 2 are all cheating!

Here's what we found 🧵

[alex zhang @a1zhang]

Ran a small eval today on an LM using GPT-5.2 as a judge. Model scores 10%, but paper reports it scoring 34%.

I see that the paper uses GPT-5.1 as a judge; for the sake of consistency I change it.

Switch to GPT-5.1 as a judge. Model now scores 43.5%...

bro

[Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) @rao2z]

Getting an early morning 35,000 ft perspective on the latest LLM literature..

[Juan Vazquez @Juan_VaGu]

@datagenproc Are there any benchmarks that stratify tasks over discrete levels of verifiability?

[Juan Vazquez @Juan_VaGu]

Thank you for digging into the data to corroborate one of our findings, and for having publish a truly transparent and open source model like olmo3!

[Hamish Ivison @hamishivi]

This was super interesting to see and cool work!

I ran a lot of the {data, training, infra} for olmo 3 RL, including decontam, so was curious about how this slipped through and dug into my records.

Note everything I'm doing is using publically available data/etc!

[Hamish Ivison @hamishivi]

This was super interesting to see and cool work!

I ran a lot of the {data, training, infra} for olmo 3 RL, including decontam, so was curious about how this slipped through and dug into my records.

Note everything I'm doing is using publically available data/etc!

[Nathan Lambert @natolambert]

Sort of research that can only exist with projects like Olmo.

Love to see these and would love it if folks keep sending them to me.

[Lucas Beyer (bl16) @giffmana]

Back in the day when we figured out that large-scale pretraining seems to be a great thing, I then very quickly decided to spend a significant amount of time and effort towards removing all our downstream transfer/eval task's images from our pretraining data. But I didn't just remove the exact images, I also removed their "semantic duplicates".

I don't really have much to show for it, only this picture in the Appendix of "Big Transfer (BiT)", which was a bit shocking: there's neardups across essentially all major datasets, including between ImageNet train and val(!) and between the 32x32 CIFAR and the much larger ImageNet(!)

We've done this thoroughly in all our works thereafter, sometimes not even mentioning it anymore, although I believe we were the only vision group in the world to properly do this. I like to tell myself that this plays a small role in SigLIP never feeling benchmaxxed and withstanding the test of time quite well, but who knows, I like to tell myself many things.

The thread/paper I quote below is a detailed investigation of this in the language domain, and it looks very well done. I believe doing this properly in language is even harder than in vision!