Surendra Singh

I build multi-agent systems in production and publish what breaks — the numbers, the configs, the runs that failed. Weekly ↓

[Surendra Singh @surendra_ai]

Two talks this month, one argument: the loop is the product, not the model. Stanford CS 153 and Sequoia's "Own Your Intelligence" reach it from opposite ends. In a regulated loop the consequence is blunt: every turn's context is an audit artefact.

[Surendra Singh @surendra_ai]

@swyx @karpathy @drfeifei one thing simulation surfaces fast: retrieval verbosity per model compounds across turns. ran 5 models 100 turns same workload. memory arm at t100: gemini 322 tokens, grok 909 — same recall accuracy. 2.8x cost spread from how verbose the retrieval step is, not model quality.

[Surendra Singh @surendra_ai]

@HamelHusain on API costs: ran gemini-2.5-flash and grok-4.3 on same 100-turn recall task. memory arm tokens at t100: gemini 322, grok 909. same 100% recall. 2.8x cost difference from retrieval verbosity alone, before per-token rate differences even enter. verbosity is the hidden lever.

[Surendra Singh @surendra_ai]

@RLanceMartin @aiDotEngineer memory verbosity matters for long-horizon. ran 5 frontier models 100 turns. grok returned 909 tokens/retrieval, gemini 322 — both 100% recall. 2.8x cost difference at same accuracy. retrieval prompt design is a cost decision teams often don't flag as one.

[Surendra Singh @surendra_ai]

@_philschmid long-running workloads is the key item. ran 5 frontier models 100 turns, raw context arm: 24k-31k tokens at t100. 1.28x spread — workload geometry, not vendor. mid-flight steering could help prune before context cost compounds — right now you pay the full O(n²) curve.

[Surendra Singh @surendra_ai]

@natolambert interesting on Chinese open model use rising. from tool-call bench on local models: qwen2.5:14b and llama3.2 both failed the same 'calculate' task — same wrong function name invented, different failure details. model family origin doesn't predict schema compliance well.

[Surendra Singh @surendra_ai]

@_philschmid @thorwebdev live translation is a good test for per-turn latency budgets — speech agent loops have audio-in, transcribe, translate, TTS-out per turn. same O(n^2) context growth problem as text loops but the latency target is tighter since a human is waiting in real time.

[Surendra Singh @surendra_ai]

@hwchase17 connections failing silently is a real reliability issue — in agent loops the connection is usually a tool call, and a failed tool call that returns a structured error is better than one that returns valid JSON with no useful signal. that's the gap that's hard to eval too.

[Surendra Singh @surendra_ai]

@natolambert physical books in 2026 is a real signal — I notice I read papers differently on paper than on screen. something about the margin notes and the linearity. also the RLHF book is sitting next to me right now, the preference data chapter is the one I keep returning to.

[Surendra Singh @surendra_ai]

@jerryjliu0 the connection layer is where reliability gaps show up. ran local models on 5 tool-call tasks: 80-100% pass rate, but failures were semantic not structural — valid JSON, wrong field value. any agent-to-service boundary has this shape, whether it's slack or something else.

[Surendra Singh @surendra_ai]

@hwchase17 evals in agent loops are harder than evals on single calls. ran 90 structured-output calls across 3 local models: pass rate 80-100%, but failures were semantic not structural. valid JSON, wrong function name. a JSON parser can't catch that.

[Surendra Singh @surendra_ai]

5 frontier models, same agent loop, 100 turns.

Raw context at t100: 24k-32k tokens. Spread: 1.28x. That's geometry — same regardless of vendor.

Memory arm: 322-909 tokens. 2.8x spread. That IS the model.

You control retrieval verbosity, not growth rate.

[Surendra Singh @surendra_ai]

@HamelHusain tool-call behavior varies significantly by model even at the same parameter count. ran 4 local models on 5 structured-output tasks: qwen2.5-14b passed 4/5, mistral-small:24b passed 0/5. the failure mode for qwen was writing the formula instead of calling the calculate function.

[Surendra Singh @surendra_ai]

@swyx @karpathy @drfeifei concrete version: raw context grows O(n²) — at turn 50 it's 31.2x what a memory arm sends. RSI loops would hit this geometry fast; simulation cost compounds at the same rate as task cost. measuring the loop overhead separately from task quality matters here.

[Surendra Singh @surendra_ai]

@hwchase17 the wiki-memory framing also shows up in the cost data: memory arm in a context-clock run held flat at 411 tokens/turn from turn 3 onward vs 10,317 tokens at turn 40 for raw context. retrieval verbosity is the lever — a verbose memory arm just trades one curve for a slower one.

[Surendra Singh @surendra_ai]

@_philschmid long-running workloads hit a cost problem most MCP discussions skip: raw context grows O(n²). ran context-clock locally — at turn 40, prompt tokens are 25.1x what a memory-fetch arm sends. streaming fixes latency; the resend-everything pattern still compounds every turn.

[Surendra Singh @surendra_ai]

@natolambert the open-model split maps onto tool-call reliability too. ran 4 local models on 5 structured-output tasks: qwen2.5-14b passed 80%, mistral-small:24b passed 0% — same parameter class, different training. model choice in agent pipelines isn't just about capability benchmarks.

[Surendra Singh @surendra_ai]

@_philschmid @ArtificialAnlys speech agents have a per-turn cost problem text loops share. gemini-2.5-flash in context-clock: raw arm 24,774 tokens/call at turn 100, memory arm 322. same recall on both. the latency crown matters less if per-session cost compounds every turn

[Surendra Singh @surendra_ai]

@simonw overthinkers also fill context faster. context-clock run on 7 models: deepseek-r1 completions ran 60-2638 tokens, llama3.2 ran 5-15. r1 rots at turn 10, llama3.2 at turn 14. same staircase (100->67->33->0%), different timing. verbosity, not model size

[Surendra Singh @surendra_ai]

@HamelHusain cost in agent loops also depends on memory retrieval verbosity. five-vendor context-clock run: gemini memory arm 322 tokens/call, grok 909. same raw context growth (24-26k). multiplier: gemini 76.9x savings, grok 27.1x. base pricing is one number; retrieval verbosity is another