sumit vohra

Agentic Evaluations @Rippling | prev Neobrix ( acq Rippling ) | Alliance DAO | YC W23

[sumit vohra @sumitvohra1]

@YashHustle_22 How lightly “it’s over for other” is used, its hilarious 🤣

[sumit vohra @sumitvohra1]

After building & evaluating AI at @Rippling for the past year, one lesson stands out:

Enterprise AI is as much about latency as correctness.

Most real use cases are just memorized clicks on loop — not one-offs. If you want users to feel "can't live without your AI", it better be faster than clicks.

The day we do that, it will be true PMF for AI assistants. 🚀

[sumit vohra @sumitvohra1]

@thejackobrien @composio @thejackobrien can u share any benchmark results of avg latency for glm compared to competitors. I want to know first if you guys are better than fireworks and baseten before i pitch your product. You can dm me.

[sumit vohra @sumitvohra1]

@jdoliner Should have used bytedance new video model.

[sumit vohra @sumitvohra1]

Search being so trivial problem to solve today is still fucked up so badly. Whether it’s mac or slack, i just cant find my stupid shit.. 🥲

[sumit vohra @sumitvohra1]

@lancefuchia @OpenAI Pretty cool!! Congrats.

[sumit vohra @sumitvohra1]

GPT 5.5 vs Opus 4.8 in our agentic harness.
Both have interestingly different personalities 🤔

GPT = Front Bencher
Organized, attentive, strictly by the book.
✅ Reliable, consistent, follows instructions perfectly
❌ Freezes on vague prompts/AOPS or edge cases

Opus = Back Bencher
Observant, intuitive, figures it out from context.
✅Creative, adaptive, thrives in ambiguity
❌ Sometimes spends too much time day-dreaming, or nails the solution but misses the marking scheme

Both beasts. Just different personalities.

Anything interesting you guys noticed❓

[sumit vohra @sumitvohra1]

@Yuchenj_UW Cant agree more, those dudes have distilled anthropic to their last drops.

[sumit vohra @sumitvohra1]

GLM-5.2 guys have definitely cooked 🔥

Quantitatively speaking, zero prompt tuning, zero fine-tuning — and it’s already performing within 1-2% from Opus on our evals. Haven’t seen that level from Qwen or Minimax yet.

Latency still hurts though :/

Chinese open-source LLMs are either absolute killers at distillation or straight-up doing crazy engineering. Either way, OS models + solid inference are about to become the default choice, not the vendor-lockin backup.

Super excited to see what new techniques people cook up for squeezing even more out of these models — fine-tuning, prompt tuning, quantization!!

Any creative or proven ideas on what’s actually working (or not)? 👀

[sumit vohra @sumitvohra1]

Hot take: If anyone says they “know how to build enterprise agentic evals”… they’re full of shit
It’s 100% WIP.

Been grinding an eval engine from scratch at @Rippling for 10 months. Solved a bunch — still solving plenty.
Here’s the real pain:

1/ Evals aren’t benchmarks anymore.
They’re go/no-go gates. One critical agent fuckup = instant release blocker.

2/ “It works on my data” is cope.
The real game is running evals on representative customer data at scale.

3/ Eval selection sucks.
How do you pick the right subset for maximum signal while staying under time/compute budgets (per PR vs per deploy)?

4/ Regression detection is a nightmare.
20+ teams shipping / 5,000+ evals gating releases = good luck spotting the real alarms.

5/ Reliable simulators are brutal.
They need to talk like humans, switch personalities, and NEVER force or leak the answer.

6/ The framework must be bulletproof.
You are solving for product-managers here. They should only write what they know best, the "agent expectation given a conversation". Whether it's auto-discovery, execution-framework, or data-setup/cleanup hooks all has to be provided for.

7/ Debugging + auto-tuning is brutal.
Why did the agent break? Can the agent self-fix its prompts/skills/tools? Especially when one commit or model swap tanks your scores by double digits.

8/ Setting up evaluators is tricky AF.
LLM-as-a-judge alone is dumb. You end up with weak rules like “is data-type(x) present?” while data drifts. Strong evaluators must verify correctness, presentation, and explainability.

9/ Reporting has to be sharply actionable.
Product teams want to know the fuckup immediately: exactly what broke, where to fix it, and how — no vague dashboards.

Will be posting more on how we are solving each one of them in detail in coming days!!

What’s your biggest eval headache right now? 👇

[sumit vohra @sumitvohra1]

With this, @Rippling is stepping into a new headless version of sass-software. We are not one of those bullshit AIs which refer you to help-center articles, we take actions, really actions and save your time!!!

[Parker Conrad @parkerconrad]

Rippling launched its AI analyst today. I'm not just the CEO - I'm also the Rippling admin for our co, and I run payroll for our ~ 5K global employees.

Here are 5 specific ways Rippling AI has changed my job, and why I believe this is the future of G&A software. 🧵 1/n

[Tripti Singh 🇮🇳 @triptidaudsar]

If you do not speak up today .. This will be the result of #aravallihills tomorrow .

#SaveAravalli

[Wisdom Walk @wisdom_walkss]

Aravalli Hills in Real Danger?

[sumit vohra @sumitvohra1]

Indian IPO market is so similar to meme-coin crypto market; both are speculation markets with no justification for their retail-price; and yes VCs are the only true winners since they are able to make billions offloading shit; that's why all those stupid posts and nonsense.

[Divya Gandotra Tandon @divya_gandotra]

Rajasthan just removed Dr. Ashok Sharma… a doctor so trusted that an entire village broke down in tears after hearing he was APO’d (Awaiting Posting Order). https://t.co/x7uZGdAScu

In a state already struggling with a shortage of govt doctors, this is the one man villagers called “irreplaceable”.

His “allegations”?

> Making reels during duty hours
> Not shutting an illegal medical shop (which the CMHO later closed anyway)

So a doctor loved for his work, communication, and dedication gets punished while actual corruption usually walks free.

This decision needs a serious rethink.

Healthcare in rural India is already fragile.

Removing a doctor the public trusts only makes it worse.

[sumit vohra @sumitvohra1]

controversial opinion: @airindia has one of the best chatbots out there 🛩️

[sumit vohra @sumitvohra1]

@parkerconrad Wohooo 🚀🚀

[Parker Conrad @parkerconrad]

In other tech news - Rippling is launching corporate travel.

https://t.co/S5BmtFPWl3

- Control travel spend before it happens ("only allow booking flights within 30% of the cost of cheapest reasonable flight").

- Because it's integrated with Rippling, you can easily build out policies based on someone's role (different policies for VPs than Managers).

Congrats @mustbesyrup , who founded Rippling Travel and has been building it for the last 3 years!