Braintrust

The observability layer for production AI.

[Braintrust @braintrust]

Topics is now GA on all plans.

Continuously find the patterns worth investigating across your production traffic.

[Braintrust @braintrust]

We ran the 48 Group Stage World Cup matchups through six configurations of @p0 research agents and scored every output in Braintrust.

Here's what we found → https://t.co/BrOi713GnE

[Izzy Hurley @iz_hurley_]

In a recent @Braintrust post, I compared GLM 5.2, served by @baseten, with Anthropic’s Opus 4.8 and Sonnet 5 on a long-context code retrieval eval. Building and digging into this eval reminded me of the importance of considering the variance in your data.

[Braintrust @braintrust]

@huggingface Read more → https://t.co/Tb0iu7IR1O

[Braintrust @braintrust]

The answer to "which model is cheapest" depends entirely on whether you're asking about cost per task or cost per success.

We took 1,781 real agent traces from @huggingface and used Braintrust to eval them for patterns.

The results show that you should pick the cheapest config that clears your quality bar, but that config is different per task family. Open-weight models can manage coding, but gpt-4.1 is still better for conversational support.

[Braintrust @braintrust]

Game, set, match on the inaugural Agent Open.

Thanks to everyone who stopped by and grabbed a paddle.

And thanks to our friends and partners who made it possible: @Modal, @Browserbase, @turbopuffer, @p0, @llama_index, and @cursor_ai.

See you at the next one.

[Braintrust @braintrust]

@baseten Read the full GLM-5.2 eval → https://t.co/cIXknXKgN2

[Braintrust @braintrust]

Run the best OSS models in Braintrust, in collaboration with @Baseten.

Call GLM-5.2 natively, eval its quality, and observe its behavior in production. Save on inference costs without compromising quality by picking the best model for your agent. Free to try through July 31.

[Braintrust @braintrust]

@baseten How to use GLM-5.2 in Braintrust → https://t.co/ixUB4Xle17

[Braintrust @braintrust]

Read more → https://t.co/vhSac8HVKg

Apply now → https://t.co/1JE5S7INL9

[Braintrust @braintrust]

The AI teams that ship quality agents put evals and observability in place early.

Braintrust for Startups gives early-stage companies production-grade observability infrastructure so they can build with confidence, no matter their size or resources.

[Braintrust @braintrust]

Watch the workshop → https://t.co/NBPQWz20jB

Get started with Topics → https://t.co/teAoNNlOXJ

[Braintrust @braintrust]

Reading traces one by one doesn't scale.

Topics automatically clusters production traces so you can identify patterns, investigate failures, and decide where to focus your eval efforts.

[Braintrust @braintrust]

Learn more in the Braintrust docs → https://t.co/sgzpTTgvvn

[Braintrust @braintrust]

Run a full chess eval without writing a single line of code using the Braintrust CLI.

- Take a CSV of chess puzzles and make a dataset.
- Write a prompt to solve mate in 2 puzzles, and upload it to the project.
- Then write a scorer that compares the output to the expected answer.

The eval found that GPT‑5 with no reasoning scored about 25% on the chess puzzles, and with low reasoning it scored about 15%.

[Braintrust @braintrust]

Read more → https://t.co/xuBbWq8xsB

[Braintrust @braintrust]

What's new:

- Smarter evals with role-based score visibility and contextual rubrics.
- Secure access to OpenAI, Anthropic, Google, and Azure without long-lived credentials.
- Faster debugging with a redesigned trace experience that surfaces the most critical content.
- Higher reliability through automatic provider failover.
- Complete audit logs for better security and compliance.

[Braintrust @braintrust]

Read more → https://t.co/HYz3y9k0GC

[Braintrust @braintrust]

Stateful agents that do real work are worth investing in, but they're also more difficult to eval. The hard part isn't scoring, it's getting the state right.

The best approach is to be deliberate about what needs to be real and what can be mocked in the eval, so you balance cost and time while still capturing state accurately.

[Braintrust @braintrust]

Read more → https://t.co/6IECFVnibF

[Braintrust @braintrust]

Cost-efficiency doesn't mean picking the cheapest model. Model choice, routing, retries, fallbacks, and escalation all determine cost together.

If you want to reduce AI cost without trading away quality, the right unit of analysis is cost-per-resolved-request, measured against the quality bar your product needs.