Barrett Lattimer

Research Engineer @asapp, exploring NLP and Dialogue Agents

[Barrett Lattimer @barrettlattimer]

I'm excited to announce JOSH, a self-training algorithm to improve dialogue agents without human feedback.

Boosting LLama 3 by 75% and gpt-4o by 8% over their zero-shot baselines on tool-calling benchmarks!

We also introduce ToolWOZ a multi-turn tool-calling benchmark
Read on👇

[Barrett Lattimer @barrettlattimer]

Safer tools for all! This is a great tool for safety and peace of mind

[Xu Li @XuLi135803]

Excited to share that ToolShield is now part of OpenHands, since v1.36.0!

In our new analyzer, ToolShield provides grounded safety experiences to a guardrail and enhances its safety awareness, cutting ASR 75–88% → 7–10%.

No more regrets after an agent wipes your whole system.

Activate it with: analyzer = ToolShieldLLMSecurityAnalyzer(llm=guardrail)

The default analyzer asks the same LLM that produced a tool call to judge its own risk, the context that causes the bad action also biases the self-assessment.

A separate guardrail alone only gets to 14–18%. By self-exploring each tool's potential for harm before deployment, ToolShield distills grounded, actionable safety experiences that further strengthen the guardrail.

Release notes: https://t.co/PytWFcsAm0

[Barrett Lattimer @barrettlattimer]

Why are sub-agents (like Explore agents) one-and-done? Wouldn't it be nicer to have them hang around as cached "experts" your main agent can double check assertions against or bounce ideas off of. It seems weird we just throw them away after one use.

[Barrett Lattimer @barrettlattimer]

Claude code loves to write verification steps it'll never do

[Barrett Lattimer @barrettlattimer]

Has anyone used Claude code + opus 4.5 for Lean? It feels like Opus 4.5’s ability to iterate here could be crazy

[Barrett Lattimer @barrettlattimer]

@bcherny Right, I have been using this, but by using "always allow" to generate the regex locally then copying it to the allowed-tools frontmatter. I didn't know if y'all had thought about an "always allow for this command" that adds the regex to the frontmatter automatically.

[Barrett Lattimer @barrettlattimer]

@bcherny slash commands are super powerful, but when building in teams they often get blocked by "allow" permissions because "always allow" is scoped to local claude settings & the regex can be nontrivial to write manually. Have y'all thought about a "always allow for /command"?

[Barrett Lattimer @barrettlattimer]

Opus 4.5 really feeling like everyone got donated an iron man suit at the same time

[Barrett Lattimer @barrettlattimer]

Why is searching for previous AI chats so terrible across all the major LLM platforms? Seems like whenever I'm trying to pull up an old conversation I had that contains important information its just lost forever lol

[Barrett Lattimer @barrettlattimer]

@Dorialexander We wrote one recently on synthetic data to increase tool calling for small and frontier models https://t.co/kaHjT6XPbw

[Barrett Lattimer @barrettlattimer]

@sam_paech What have you found is the best proxy to measure “creativeness” past pure human evaluation

[Barrett Lattimer @barrettlattimer]

@OpenAI OpenAI plays Pokémon?

[Barrett Lattimer @barrettlattimer]

Why @huggingface deleted the step() function from their PPO implementation is beyond me 😂

[Barrett Lattimer @barrettlattimer]

I will be releasing some code soon to do PPO on multi-turn dialogue with sparse rewards!

It was really an interesting journey that all came back to rewards. Spoiler: the trick was to add a small formatting reward to force adherence to ReACT (similar to findings from R1)

[Barrett Lattimer @barrettlattimer]

Great announcement today. Heres how we already made it better months ago:

- Use the JOSH algorithm to improve agentic tool use on frontier models for multiple benchmarks including tau-bench.
- JOSH uses test time scaling to produce its own training data.
https://t.co/kaHjT6XPbw

[Anthropic @AnthropicAI]

Claude 3.7 Sonnet is a significant upgrade over its predecessor. Extended thinking mode gives the model an additional boost in math, physics, instruction-following, coding, and many other tasks.

In addition, API users have precise control over how long the model can think for.

[Barrett Lattimer @barrettlattimer]

Robust Function Calling (https://t.co/UE8lymaKpI)
- Function calling performance varies due to bad naming conventions
- Introduce Hammer, uses specialized dataset for irrelevance detection
- Performs consistently well across FC datasets

[Barrett Lattimer @barrettlattimer]

Here is my list of favorite ICLR 2025 LLM Spotlight Papers with bulleted notes and links
👇

[Barrett Lattimer @barrettlattimer]

Pref. Optimization for Reasoning with Pseudo Feedback (https://t.co/LDiKjwRJiD)
- Problem: Selecting correct reasoning approach for preference optimization is hard
- Use Frontier LLMs or self-consistency majority vote to create preference pairs
- Large gains in MATH and coding