Braintrust

Active observability for agents in production.

[Braintrust @braintrust]

Your team deserves more from your agent observability platform - a single, connected place for instrumentation, investigation, and measurement, enhanced with intelligence.

In Braintrust, you can start with an open-ended prompt about agent behavior and carry the investigation into code, evals, automation, and monitoring.

New tools like Patterns and Debugger make it easier to find recurring behavior, get to root causes, and act on what you learn. The context you build in one step automatically follows to the next. Work in the platform with an enhanced Loop experience, or move to your coding agent of choice.

Read more → https://t.co/ACHy6mFkuJ

[Braintrust @braintrust]

@ Jev, Clef, Luna, Mammoth-1: is this content harmful?

We tested decision models, standard LLMs, and specialized safety classifiers on thousands of human labeled examples from three public safety benchmarks to see how well they recognize harmful content.

Jev stood out for its performance at low cost. Mammoth-1 raised the fewest false alarms on safe answers among the decision models. Luna caught more harmful answers than Jev but was 3x as likely to flag false alarms.

Read more → https://t.co/7CXm9yuHgE

[Braintrust @braintrust]

Which AI assistant is the best personal shopper? It depends on how you shop.

Dots is best if you care about about price, Muse is the most careful product researcher, and Instinct is the most well-rounded across tasks.

Read more → https://t.co/5iaDnFoMXy

[Ankur Goyal @ankrgyl]

We now automatically partition and summarize trajectories when you view them in Braintrust.

It's surprisingly cheap with the most affordable LLMs, and makes it much easier to navigate long multi-turn traces.

[Braintrust @braintrust]

Bubble recently launched a completely new version of its core app-building Agent.

Braintrust helped make it 1.5x more accurate than its predecessor at visually representing user prompts, while also performing faster and more cost efficiently.

Read more → https://t.co/8EFJSY2xL8

[Braintrust @braintrust]

Phrase search with no matches.

We tested Nitro against other databases.

See the benchmarks → https://t.co/sNH6Vo3VzJ

[Braintrust @braintrust]

What's new:

- Trace your team's coding agents to track costs, usage, and recurring issues
- Review agent requests, actions, and responses in a chat conversation style thread view
- Prioritize the most important issues in your traces with Patterns severity labels
- Send alerts from Loop automations directly to PagerDuty
- Ask questions about your dashboards and build new charts with Loop
- Trace Cursor agent sessions from the CLI or desktop app

Read more → https://t.co/sZ5ZeFqccl

[Braintrust @braintrust]

Run evals at half the cost by using AI provider batch APIs in Braintrust.

Large regression suites, model comparisons, and scoring jobs can be batched at a discount if your deadline is flexible.

Submit the requests to the provider, collect the results when finished, and track everything in one Braintrust experiment.

Now available for both TypeScript and Python → https://t.co/e3fdH8lNr6

[Braintrust @braintrust]

Export your OpenTelemetry traces from the Claude platform directly to Braintrust.

Measure production behavior, eval agent quality, and continuously improve performance.

Read more → https://t.co/ZDhR5KnhK8

[Jess @daRubberDuckiee]

being able to use Codex to automate:

text into Muse -> grab results -> save them to Braintrust -> continue until done with all 25 prompts... insane.

[Braintrust @braintrust]

We gave AI assistants Muse, Instinct, and Dots the same personal shopping tasks and let them loose on the web.

We evaled their product recommendations on response quality, adherence to budget, and how often they recommended the same products.

Instinct performed best overall in our eval, but there are tradeoffs depending on how you shop. Dots stood out for finding good options at lower prices, while Muse was best for shoppers with strict guidelines.

Read more → https://t.co/T5owNyKmCR

[Braintrust @braintrust]

Send your application logs to Braintrust.

Connect app logs to agent traces to understand what went wrong and why. Debug with the context from both in one view without switching tools.

Learn more → https://t.co/FIVZjoRX7c

[Braintrust @braintrust]

Trace, inspect, and improve your coding agent's work.

See what's driving costs, find recurring problems across your team's sessions, review suggested fixes, and measure the impact of your changes.

Coding agent insights is now in public preview.

Read more → https://t.co/k8xEORIOWK

[Braintrust @braintrust]

Understand where your agent went wrong with the updated thread view in Braintrust.

Follow your agent's trajectory as a back and forth chat conversation, with tool calls and sub-agent work in context. Works for complex and custom agents built with any framework.

Now available in public preview.

Read more → https://t.co/KZTYfEUP6d

[Braintrust @braintrust]

We ran evals on Opus 5.5 and new GPT-6 models to assess both problem solving and writing quality.

- Opus 5.5 had the strongest observed results, but it comes at a cost.
- GPT-6 Luna is the standout on value, and is best for running an agent at volume.
- GPT-6 Sol balances writing, task performance, and latency, without Opus prices.

Read more → https://t.co/rgxqkomnfi

[Braintrust @braintrust]

What's new:

- Get answers about your agents faster with Nitro, Brainstore's new asynchronous query engine
- Trace Jev calls from your application or use Jev as a scorer in Braintrust
- Send application logs to Braintrust and inspect them alongside your traces
- Run Python and TypeScript evals at a discount through provider batch APIs
- Group projects into named collections for repeatably assigning access to specific users

Read more → https://t.co/58RL6bHkh6

[Braintrust @braintrust]

zyBooks builds interactive textbooks. Patterns helps resolve student support requests faster by automatically surfacing issues for their platform engineering team.

Try Patterns → https://t.co/NJqaSj9lZ4

[Braintrust @braintrust]

We evaled the @Resend and @Postmarkapp MCP servers across a range of common tasks to see how agents perform when automating transactional emails.

Postmark returned acceptance confirmations faster on average. Resend had higher first-lookup success. Both passed all seven deterministic scorers across the 90 tasks we tested.

The choice comes down to your specific workflow, since each provider offers features the other does not.

Read more → https://t.co/o2Op6jh69p

[Braintrust @braintrust]

How does Patterns find failures you haven’t written evals for? Can it catch hidden bugs? How do you know a fix worked?

Patterns, Topics, and Loop help you find agent failures, turn them into evals, and track improvements.

Read the FAQ → https://t.co/WbsPBDr7UO

[Braintrust @braintrust]

GPT-6.1 Sol promises similar performance to last week's GPT-6 release at one-fifth the token cost of Astra. Does it deliver?

We re-ran our eval on how well models can guide users through multi-step problems. GPT-6.1 Sol reached 78.6% problem-solving accuracy at token prices far cheaper than Astra, Opus 5.5, or Fable 5.1.

The most notable improvement was how the new model guided users through teaching interactions. Our "teaching quality metric" for GPT-6.1 Sol rose from 64.1% to 77.6%, compared to Astra's 82.2%.

The results indicate GPT-6.1 Sol is able to interact with users in the ways they specify, leading to better pedagogy performance.

Read more → https://t.co/iUPInldRIt

[Braintrust @braintrust]

GetYourGuide is the online marketplace for finding fun activities around the world. Patterns surfaced a failure buried in their logs and made it actionable.

Try Patterns → https://t.co/NJqaSj9lZ4