Joachim Baumann

Postdoc @StanfordNLP @StanfordAILab / Prev: @MilaNLProc @UZH_en @MPI_IS @CarnegieMellon. CompSocSci, LLMs, algorithmic fairness.

[Joachim Baumann @joabaum]

We present SWE-chat: the first large-scale dataset of coding agent interactions from real users in the wild.

In 40% of real coding sessions, the agent writes ~all the code. Users push back 39% of the time – agents almost never stop to check.

Data, paper, & findings in the 🧵👇

[Tiancheng Hu is looking for a job @tiancheng_hu]

Did you notice something missing in recent LLM releases? GPT-4 featured detailed calibration curves. Since then, they've vanished from almost every major LLM technical report. Our #neurips2026 position paper argues this is a mistake and we must embrace calibration from training to evaluation to deployment. Here’s why:

[Justin Lo @loresjpg]

Nice DeepSWE score! Does it survive a lowercase filename?

Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.

Models clustered around 74% on DeepSWE end up 23 pp apart.

Good at coding, or good at DeepSWE? 🧵

[Joël Niklaus @joelniklaus]

I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.

Which harness should you run your open model in? I ran 28,849 trials to find out.

The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.

The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.

Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.

One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.

The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.

@opencode costs $3.35 per solved task, 11.7 times as much as Pi.

See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.

Leaderboard: https://t.co/AiWiQtBOz5
Harness Lens: https://t.co/I8jbcWa9JH

[Michael Ryan @michaelryan207]

Some of you know I’ve gotten deep into teaching recently, so I was thrilled to have the opportunity to sit down with Peter Norvig (one of the best AI educators to ever do it!!!)

Peter co-wrote THE textbook on AI!

In this discussion we talk about what is still important for humans to learn and do when it comes to AI development, and how we should think about setting the objectives for AI!

Check it out! ⬇️

[Diyi Yang @Diyi_Yang]

Thank you @LaudeInstitute for the generous support!

So excited to work with @tatsu_hashimoto, @chrispiech, @erikbryn and our team to build collaborative AI that complements human expertise and helps people develop new skills as work evolves!

[Laude Institute @LaudeInstitute]

Collaborative Intelligence for the Future of Work | @Stanford

@Diyi_Yang, @tatsu_hashimoto, @chrispiech, and @erikbryn are building AI systems that complement human expertise to equip the workforce of the future.

https://t.co/D759wobD9w

[MilaNLP @MilaNLProc]

A few snapshots from a brilliant week at @emnlpmeeting in Budapest! Lots of good conversations, old friends, and new connections!

[Jennifer Wang @jennjwang]

Jobs are modeled as bundles of tasks. But as jobs change, how do we keep those task lists up to date?

Workers often experience new responsibilities and tools before they reach their job descriptions.

We start by listening to them. 🧵

[Joachim Baumann @joabaum]

@kinggary Interesting setup! In my opinion, questions of diversity, gameability, and algorithmic monoculture remain, even if AI reviewing is getting better. Our ICML paper might be of interest: https://t.co/Fo8WN0R8mP

[Joachim Baumann @joabaum]

Can you boost your AI review scores by asking an LLM to rewrite your paper?
Yes! We call it paper laundering
Our @icmlconf spotlight paper argues current AI reviewers aren't ready to automate peer review, and outlines what a science of peer review automation should look like🧵👇

[Tiancheng Hu is looking for a job @tiancheng_hu]

Time flies and I am looking for full time research positions! I am finishing my PhD at Cambridge early 2027. My research focuses on simulating human behavior with LLMs, LLM personalization, evaluating and shaping model behavior, and understanding human–AI interaction. Please get in touch if you are hiring in relavant areas!

[Joachim Baumann @joabaum]

@nathanhabib1011 @huggingface looking forward to what you'll find! and would be great to add a traces tag (I think the problem is that raw traces live in the transcripts folder)

[MilaNLP @MilaNLProc]

Our lab @MilaNLProc has 6 papers accepted at EMNLP and workshops! 🎉
See you in Budapest 🇭🇺

[Debora Nozza @debora_nozza]

📢 I’m hiring!

Excited about NLP & Computational Social Science? I have an open 2-year postdoc at @MilaNLProc on personalized & subjective NLP, focusing on everyday language technology use and their social & ethical impact

📅 Deadline: Nov 15, 2026
🔗 https://t.co/04JYceRYCe

[Ricardo Olmedo @rdolmedo_]

With only ~$1,000 of training compute, we trained 𝗳𝗿𝗼𝗺 𝘀𝗰𝗿𝗮𝘁𝗰𝗵 a 1.7B coding agent that beats Claude 3 Opus and GPT-4o on SWE-bench Verified 🤯

At 25.5% pass@1, it sets a new record for nanoswe, our open SWE-bench speedrun.

How is this even possible? 🧵👇

[clem 🤗 @ClementDelangue]

We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see.

If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!

[Weixian Xu @WeixianXu]

Language models already excel at math, but human growth remains essential: we need AI to teach people, not replace them!

Teaching is different from solving. As the Chinese principle “因材施教” suggests, preferences differ from person to person, and a good teacher should adapt their teaching to each learner.

We introduce Sherpa, a framework for training LLMs to teach adaptively.

[Hayoung Jung @hayounggjung]

👀 Today, we release our live evaluation dashboard for SciConBench: https://t.co/GZg2UtJ3yb

Frontier models keep getting released every 1-2 weeks with claims of huge capability gains.

But are they actually getting better at synthesizing scientific conclusions in health?

So far: surprisingly little.

We’re going to keep measuring the performance of the *latest* frontier models on this high-stakes, real-world task over the next 1-2 years. Check out the thread here↓

[Caleb Ziems @cjziems]

I'm on the academic job market this year!

Recently I've been working on scaling laws and human-centered interventions for building more socially grounded AI.

If you're attending #COLM2026, please drop me a DM or email. I'd love to chat!

[Vishakh Padmakumar @vishakh_pk]

I'm on the job market this year!

I work on measuring the societal impacts (overreliance, homogenization etc.) of AI/LLMs and proposing ways to mitigate these.

I'm at #COLM this week, and #NeurIPS later this year, and would love to chat more!