Sandeep Bele

Founder, . Agent evaluations.

[Sandeep Bele @sandeepbele]

@1752vc Fastest way to start with agent evaluations - https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

Build agent evals directly from coding harness - https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

Introducing https://t.co/Z2lrlPiUEY 🚀

Build agent evals directly from coding harness. Instrument agent, create rubric, simulate agent-user interactions, run evals just by prompting.

link 👇@HaliosAI

[Sandeep Bele @sandeepbele]

https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

Introducing https://t.co/Z2lrlPiUEY 🚀

Build agent evals directly from coding harness. Instrument agent, create rubric, simulate agent-user interactions, run evals just by prompting.

link 👇@HaliosAI

[Sandeep Bele @sandeepbele]

@hp_arora Hey thanks. Sure will try!

[Sandeep Bele @sandeepbele]

Been reading about Jev from TypeSafe AI. The confidence/probability angle is interesting,

Some use cases I would like to test once I get access -

How useful is a decision model if you don’t get reasoning behind the decision?

How well do the probabilities hold up outside the domain it was trained on?

And how sensitive is it to the way a question is framed?

For example, in a RAG eval, the context has two refund policies, one outdated, and the agent answers using the stale one. What probability or confidence does Jev assign there?

I can already see some interesting uses in @HaliosAI for PII/secrets detection, guardrails, and model routing.

[Sandeep Bele @sandeepbele]

@jeffboudier @huggingface https://t.co/Z2lrlPiUEY - lets devs build evals for their agents directly from coding harnesses.

[dax @thdxr]

be careful in your pursuit of cheap tokens

we've come across so many sketchy things that are out there with tons of usage

remember you're hooking up your harness to these inference providers. they can and do crazy things like send fake tool calls to steal your info

they also sell entire traces to third parties who resell again. a single key in the trace is enough to screw you

if you cannot explain how someone can offer cheap tokens you should avoid it

[Wulfie Bain @wulfie_bain_]

I lead OpenAI's Applied AI Engineering for startups internationally, & I've seen even the best startups end up with bad prompts. We've then fixed them.

We've sped up some agents by 50%, & saved others $ millions in LLM spend.

Here's how: https://t.co/TzH1BJG3ab

[dax @thdxr]

if you are a programmer and you'd like to start a company it is helpful to internalize two big weaknesses you have

1. your instincts for anything outside programming are terrible. everything you believe is likely 100% backwards

2. you've always been treated like the smart person in the room so it will take you a very long time to understand #1

for me personally it took ~ 10 years before i started to be able to unwind this

[Sandeep Bele @sandeepbele]

@boristane try https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

@michael_chomsky if you want to catch agent behavior failures : https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

👉 https://t.co/Z2lrlPiUEY

[Sandeep Bele @sandeepbele]

claude code can write your eval scripts. that part is not really hard anymore.

but you still need somewhere to run simulations repeatedly, keep the results, put them in ci, bring in production traces, figure out why something failed, let other people work on the evals, make the same setup work for the next agent...

and now those few scripts are internal eval infra that somebody has to own.

maybe you want to build that. if not, that is basically why we built Halios.

[Nikesh Arora @nikesharora]

Summary:
I spent time trying to figure out this orchestration layer problem, can we design a multi model architecture in the long term. The more I dug in the more I understand that trying to build an abstracted layer is hard. As agentic activities increase and agent chaining and complex tasks get assigned to AI it will become harder to move between models.

There is a reasonable probability that 75% of the enterprises will build their implementation of the solution to their core problem around one model "stack".

Token price reduction by 90% is the solve and mobility between models from the same frontier lab!

Evals, harnesses, cache memory are the moats and I don't see models providing simple abstraction to those.

I know there are efforts to do this out there, the long term solve for orchestration if it works will need to be "Claude code" level of design genius.

Here's a chat with Fable @HamzaFodderwala had.

**Why abstraction looks easy.** Models are stateless — every API call is weights + a prompt assembled at runtime. Everything the model "knows" about you — memory, documents, history, tools — is injected into the context window by software outside the model. So in principle, all your state already lives outside the weights. The catch is what "state" includes.

**Layer 1 — Data (fully portable).** Enterprise documents, tickets, logs. Retrieved via RAG: text is chunked, embedded, stored in a vector database (Pinecone, pgvector), and relevant pieces are fetched into the prompt per query. The embedding model is separate from the LLM, so this layer is genuinely model-agnostic. Already solved.

**Layer 2 — Memory (portable in principle).** Systems like Mem0 and Zep sit between the app and the model: after each interaction they extract salient facts ("user prefers X"), store them as plain text, and inject the relevant ones into future prompts. Because the artifact is natural language, it reads into any model. Facts port.

**Layer 3 — Orchestration/routing (works, but only for shallow tasks).** Gateways like OpenRouter and LiteLLM normalize API differences and route each request to the cheapest capable model. This is the fungibility layer being furiously built. It genuinely works for one-shot, verifiable tasks — classification, extraction, summarization — which conveniently are the tasks where cheap models suffice anyway.

**Where it breaks — the non-portable state.** Four things stay behind when you switch:

- **The harness.** Prompts, tool schemas, and guardrails are tuned to one model's quirks. An agent must get every step right, so reliability compounds: a model that's 98% reliable per step completes a 50-step task about a third of the time; at 90% per step, it almost never finishes. Swapping models costs you a few points per step — the difference between an agent that works and one that doesn't.
- **The evals.** Swapping means re-testing everything and re-fixing every regression. The real switching cost isn't data migration — it's re-verification. Nobody has abstracted that.
- **Procedural memory.** Facts port; skills don't. Cached successful workflows and learned workarounds are conditional on the model that produced them.
- **Cache pricing.** Provider-specific, worth 75–90% of input costs on agentic workloads. Quiet lock-in.

**The labs' angle.** They offer hosted memory, hosted file stores, caching, fine-tuning — every one pulls state from your side onto theirs. The labs will crack memory first, but as lock-in, not portability. Nobody standardizes their own exit door. MCP is the partial exception: it standardizes tool and data access across models, but doesn't touch harness tuning or evals.

**Where 3P vendors fit.** Routers are thin-margin commodity plumbing; vector DBs and memory infra are real but small. The two structurally interesting positions: **eval platforms** (LangSmith, Braintrust) — since switching cost equals re-verification cost, whoever industrializes cross-model testing actually enables fungibility.

[Sandeep Bele @sandeepbele]

This is great!

[National Design Studio @ndstudio]

Today, we are releasing Rampart: a 14.7MB machine learning model designed to protect citizens’ privacy by redacting personal information directly in your browser before it gets sent to any server

[Sandeep Bele @sandeepbele]

Consumers are subsidized by labs because of their data. Enterprises pay premium because (hypothetically) data stays out of training. Will companies share their data for cheaper tokens?

[Nikesh Arora @nikesharora]

The AI Business model trap: LLMs want cash flow to fund the race to AGI or the next model. Enter free consumer AI - they are losing a lot of money on the breadth of models to serve consumers for free! They are caught in the post training data trap, free consumer usage feeds post training needs, it can't be right to stop serving customers for free?

But they need money for the compute:

The monetization challenge is being pointed to Enterprises.

Phase 1 - seemed easy, value capture in coding, the most bottom up motion in enterprise - with low customization per customer. Developers continue to train coding, tasks and eventually will train flawless skills.

Phase 2 is where the challenge lies, showing true enterprise value. The promise of efficiency, accuracy, elimination of resources - that requires a different approach, build depth with harnesses, context, memory, solving for edge cases with deterministic guardrails! Build skill libraries - enter FDEs. Yes,FDEs will train the enterprise Waymos of the world.

The risk - high token pricing for enterprises while consumers for free! Yes for consumer distribution businesses (aka Google, Meta, Apple, etc) it makes sense to hold on the distribution with free AI.

If you want to win enterprise, you should be forward pricing tokens. The cheaper the tokens for enterprises it will allow for experimentation, workflow reimagination - instead CIOs are busy restricting AI use and working on making the use more efficient!

Paradox: They still haven't fully understood and embraced the value of AI in the enterprise.

If I were them:
1. Cut token pricing now, else send enterprises to secure opensource and end up with friction filled routing layers.
2. Show me how enterprises can use their context, training and data as their competitive advantage.
3. Build tools for rapid edge case learning and reducing false positives.

@HarryStebbings
@sama @DarioAmodei @demishassabis

[Sandeep Bele @sandeepbele]

between this and possibility of new coin loaded WhatsApp .. my emotions are torn 😭🤣🤣

[Kunal Shah @kunalb11]

It’s been a minute.

2015–2018
- Exited FreeCharge. Spent time learning and investing.
- Pondered about: Why can't trust be rewarded? Started with $1M of personal capital.
- Launched CRED to reward people for paying credit card bills on time.

2019–2025
- Built a system run by a team that values ownership, judgment, and craft.
- Grew from 0 to 17M members by aligning incentives with behaviour.
- Built several products during COVID lockdowns.
- Raised $900M+ from global investors. Did 4 ESOP buybacks.
- Made Indiranagar and IPL ads slightly more interesting.
- Received a full stack of regulatory licences.
- Lost 35 kilos.
- Scaled from 0 to ~$325M ( ~₹3,200 crore) in annual revenue across payments, lending, insurance, commerce, wealth, and credit cards.
2026
- First profitable quarter (yet occasionally asked what our business model is)
- Raised another $900M from Meta in primary and secondary capital.
- Announcing our 5th ESOP buyback.

Today

CRED is ready for its next phase. I am stepping back and @miten steps in as interim CEO, partnered with an incredibly talented team. He has been heading strategy and finance and suffering me since 2020. I’m stepping away from the operating role and will continue as a shareholder. My commitment doesn’t change. Just the role.

Extremely grateful to our members, partners, regulators, and investors who made this possible. And to our board, Shailendra, Micky, Saurabh for their extraordinary conviction.

Team CRED, I’ll still expect you to be a 10x version of yourselves.

As for me, I’ll be joining Meta to lead WhatsApp globally.

Meta comes in as a minority investor in CRED. No access to member data.

While it’s come very far, the delta between WhatsApp today and its full potential is massive. I look forward to working with Mark, Chris, and the leadership across Meta for the next step in WhatsApp’s journey. Will, thank you for scaling something the world relies on quietly, and for making this transition smooth.

Onwards.

[Satya Nadella @satyanadella]

https://t.co/vLmiBKTtX3