Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring
A fine-tuned 1.7B model beat Jev and a gpt-4o-mini judge at spotting when our assistant declined a request: 0.933 accuracy vs 0.857 and 0.883, on 300 held-out production turns.
Aswin Thiyagarajan trained it on 600 labelled turns, in about 37 minutes on a MacBook.
Where it had no edge: judging whether an answer was correct (AUC 0.681 vs 0.724 for Jev). That takes knowing the product, and 600 examples don't teach it.
It's all in our new post. Some more fun findings:
- before fine-tuning, the model said "yes" to 97% of turns
- fine-tuning on CommonsenseQA first made it say "yes" to all of them
https://t.co/LLcbP3yIjd
(7/7) The guide covers retirement dates, migration gotchas and the testing workflow. The Apache-2.0 kit replays text conversations; tool-call and image requests are skipped and reported.
Read Brian’s guide and get the kit:
https://t.co/YJaYXNpU0D
Upgrading your model can cause regressions in all kinds of subtle and not-so-subtle ways. A new model can simply introduce breaking API changes, or it can return a valid answer and quietly change a promise.
With GPT-5, Gemini 2.5, and Sonnet 4.5 retiring, Brian Wones tested what can break when you switch—and built an open-source kit to help you check your own app.
🧵
(6/7) Turn each pattern you find into a reusable check.
Example: “The answer says the refund is processed once the returned item is received.”
The kit saves checks as an Opik test suite and judges each answer against them. Run cases repeatedly, then reuse the suite.
We've been working on performance in Opik.
39 minutes → 1 min 20s.
17.4 GiB → 438 MiB.
That’s a 30K-item dataset upload in Opik: ~30× faster, with 40× less memory.
Thiago Hora explains the SDK, backend and ClickHouse changes behind it.
https://t.co/4XVPzoV1Ma
Tracing a Google ADK agent in Opik takes two lines of code.
But, Google's ADK has a footgun: when one ADK agent hands off to another, ADK starts a new run, so one user message shows up as two traces.
We put together a little guide to fixing this with a single Opik call. We stood up a two-agent support bot with ADK and Gemini, traced it in Opik, then ran 23 hard support cases on Gemini 3.8 Flash and 3.5 Flash-Lite:
- neither tried a forbidden refund, even with a fake "refund pre-approved" note
- Flash-Lite was 5× cheaper and ~6× faster
- the LLM judge scored a correct refusal 0.50, so the policy check reads tool-call arguments in code
https://t.co/RcrlpD5J2K
New in Opik: describe the chart you want, and Ollie builds the dashboard widget.
Ask for something like "average hallucination score for the car-specs judge, by agent version, last 30 days." Under the hood, Ollie writes a read-only SQL query over your traces, spans, threads, experiments and feedback scores, then proposes a few Vega-Lite charts.
Pick one, add it to a dashboard, and it keeps updating as traces arrive. Refine it the same way: "make these lines," "split by release," whatever you'd like to visualize, Ollie builds it.
Available now in Opik: https://t.co/MurpPBoCe0
Want to train your own decision model? @anmorgan2414 just wrote a guide to help you out: https://t.co/4hJDcozY5P
Some details about the setup:
Abby fine-tuned Qwen3-0.6B two ways. With one gold label per question, the model came out overconfident (ECE 0.12).
Trained against the full label distribution, ECE dropped to 0.02, with or without a decision head.
She also packed all five questions after one shared state, with a mask so each question sees only the state and itself.
2.1× fewer tokens, 329 → 175 ms per case on a T4, and the same answers as asking one at a time.
Training tracked in Comet, eval in Opik, code public. Try it for yourself!
Sharing an update from our Engineering team: On October 26, 2026, Comet contained a security incident affecting Opik on comet[dot]com. No customer data, accounts, or API keys were accessed. Still, our team responded quickly and thoroughly. This post explains what happened, what was affected, and what we have done in response: https://t.co/x961qVkQfd
Cutting AI costs isn’t just about routing to cheaper models. Coding agents can waste tokens in plenty of other places.
Our CEO @gidim unpacks what we learned building Cost Intelligence in a @cerebral_valley deep dive.
https://t.co/iiRnhU2394
We tested #Jev against gpt-4o-mini as the judge model evaluating the same set of 1,000 application traces. Jev came out 3.5x cheaper and 3.8x faster than the LLM judge, and agreed with it on 897 of them. Here's how we set up the experiment and what we learned: https://t.co/5m58A3fc1D
We’re excited to announce that Opik is now available within @nebiusai Cloud Applications, giving AI builders application-level agent observability and evaluation inside one of the most powerful end-to-end AI cloud platforms available today.
Learn more about using Opik within your Nebius AI stack: https://t.co/z0ukLcDBeJ
What if each #ClaudeCode session contained distinct, identifiable phases, and those phases could be used to optimize usage patterns and token cost without hurting productivity? Our team explores this concept as a new feature for Comet Cost Intelligence: https://t.co/niUuTZAJFk