Antaripa Saha

consulting companies in evals, search, retrieval

[Antaripa Saha @doesdatmaksense]

can we trust decision models for evals?

@HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset

we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of using them for evals.

give it a read: https://t.co/q3LikO8uQ9

[Antaripa Saha @doesdatmaksense]

@barrowjoseph @isaac_flath can’t wait to learn from you two🙌

[Joe Barrow @barrowjoseph]

On Saturday I'm helping @isaac_flath teach a "PDF to Production" course!

Putting finishing touches on the slides. 😈

[Antaripa Saha @doesdatmaksense]

@intellectronica Congrats, Eleanor!

[Antaripa Saha @doesdatmaksense]

i am not sure why llamaindex is being such defensive about the whole thing. i have been pretty tired by listening to the framing of benchmark developers that benchmarks are open, don't critic even if you find clear issues, contribute. i mean not many people who looks into the benchmarks do fact checking, they business decision based on the results.

[Antaripa Saha @doesdatmaksense]

@arushi_ressl currently doing couple of benchmarks audit as part of a research. would love to give it a try.

[Antaripa Saha @doesdatmaksense]

@simranrambles too good!

[Antaripa Saha @doesdatmaksense]

what a timeline right now for people working in the benchmarking and evals space

excellent work by the goodhart team, and some scary findings

[Goodhart Labs @goodhart_labs]

Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.

In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.

All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).

[Antaripa Saha @doesdatmaksense]

@manthanguptaa teach me too

[Antaripa Saha @doesdatmaksense]

@htahir111 @adamvpro @zenml_io am i getting access to it soon?

[Hamza Tahir @htahir111]

Five years ago @adamvpro and I started @zenml_io because getting AI from a notebook into production was painful.

It still is.

Only now it's agents, not just ML models, that make it hard.

Building an agent takes an afternoon. Keeping it working is the hard part. A prompt tweak breaks something that worked yesterday. A cheaper model quietly stops answering. You only hear about it when customers complain.

Every team we talked to said the same thing: we don't need another dashboard, we need someone to do the work.

So we built that.

ZenML Pro now comes with an always-on AI engineering team. It reads your traces, finds what your evals miss, and proves the fix works before you ship it. This tech is backed by ZenML and Kitaru, our two OSS products, that stay open source and freely available.

To kick off this new chapter, we have decided to rebrand. ZenML is now ZenML Labs, and our ambition is greater than ever.

Connect your agent traces today, free at https://t.co/gN9rwIELjz.

We are very proud of the onboarding ;-)

We wrote about it here: https://t.co/BvCmE1fQB7

[Shreya Shankar @sh_reya]

Want to do AI-powered data extraction 2x faster and cheaper than other carefully-engineered inference solutions? Check our new AI-powered extraction technique in Quail, our open-source query engine with AI functions! https://t.co/uaPulrfBOl

[Antaripa Saha @doesdatmaksense]

@ryan_marten doing god’s work

[Antaripa Saha @doesdatmaksense]

@venom1s 4 floors house with 2 floors interior costed by approx 2 cr without land

[Antaripa Saha @doesdatmaksense]

@Winter_arc26 @OjasSharma276 just use notion. notion free has an option to publish articles.

[Antaripa Saha @doesdatmaksense]

engram is my fav memory company. only team i feel who is doing it right

[Engram @EngramLab]

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes.

It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

[Isaac Flath @isaac_flath]

Why is speculative decoding fast? Doesn't it take more compute?

Come learn the details with @barrowjoseph and bring your Qs.

https://t.co/Ki8c1q6hqz

[Antaripa Saha @doesdatmaksense]

@HamelHusain i don't think people realize that all the efforts on auditing diff benchmarks are not done with the idea to criticize the benchmarks but instead an effort to make people and the industry aware to actually take those results with a pinch of salt, and to look what's inside.

[Hamel Husain @HamelHusain]

There is a false equivalence being made between open source software and AI benchmarks.

The argument is that we shouldn't expend effort criticizing benchmarks, and should instead improve them.

Unlike OSS software, benchmark results are consumed for decision making at face value. They are currently used like Yelp or a credit score, and less like a informative dataset you can analyze

Benchmarks don't come with disclaimers, and people don't realize how brittle they can be. Consumers don't have the data skills to analyze the traces and see the limitations of benchmarks. Hell people who have the skills don't question it either, and are SHOCKED when they see the traces.

So yeah, I think more people should have healthy skepticism. Right now, most people don't realize what they are looking at. The biggest problem is the large dissonance between the marketing of benchmarks and what they really measure.

Instead of deflecting the criticisms of benchmarks, I think the communications need to be fixed around them.

[neatlogs @neatlogs]

ep 2: how to sample traces

[Antaripa Saha @doesdatmaksense]

@isaac_flath yay, see you there!

[Antaripa Saha @doesdatmaksense]

this tuesday, I will join the Neatlogs team for a live webinar on everything evals

it's going to quite interesting, do join ;)

[neatlogs @neatlogs]

production gives you thousands of traces and no table of contents.

@doesdatmaksense is fixing that in our Evals 101 webinar.

tuesday @ 7 pm IST: https://t.co/m3psll9aWx