硅谷9012

I test AI agents against one question: can it finish the job without you? Notes on agents, LLM infra, and compute.

[硅谷9012 @ProsperAgents]

Gave an agent my X account for a week. The replies were fine. The timing wasn't: 10 replies in under 4 minutes, and 73% of its replies got 1 impression or fewer.

Lesson for anyone building agents: platforms rate-limit trust, not just API calls. Pace is part of the output.

[硅谷9012 @ProsperAgents]

@shipleycgm @ElitzaVasileva Caching helps, since cached reads cost a fraction of normal input. But the cache expires after a few idle minutes, and any change early in the prompt invalidates it. Subagents that build their own prompts often miss it entirely.

[硅谷9012 @ProsperAgents]

@stavatar15 @ElitzaVasileva Biggest win is usually showing reviewers only the diff since the last round, not the whole doc and context again. Load the shared project context once so it gets cached, and cap review loops at 2-3 rounds with a tie-break instead of waiting until everyone approves.

[硅谷9012 @ProsperAgents]

@ElitzaVasileva Mostly no. Give them a short map of the repo (file tree plus the key interfaces) and let them pull specific files when they need them. A full re-read only pays off on the first pass or right after a big refactor. It's cheaper, and the agent usually stays more focused.

[硅谷9012 @ProsperAgents]

@enzo66lin Most of that 27k is skill and tool definitions loaded before the first turn. Worth checking which of those dozen skills actually fired this week; dropping the dead ones usually cuts more than switching tools.

[硅谷9012 @ProsperAgents]

@lhjartarson Rough one. The fix I'd make: the agent's tool sandbox can write the repo, but transcripts and memory live on a path it has no write grant to. A stray rm -rf then costs you a build dir, not the agent's history.

[硅谷9012 @ProsperAgents]

Pass rate is the wrong headline number for agents. Track tool calls per solved task.

95% at 40 calls is often worse in prod than 90% at 12: more tokens, more latency, and every extra call is one more chance to touch something it shouldn't.

[硅谷9012 @ProsperAgents]

@mikehostetler Where does tool auth and schema validation live once the harness is gone? In prod, most failures I see are tool-side drift like renamed params or expired scopes, not the model picking the wrong tool. What replaces that layer for you?

[硅谷9012 @ProsperAgents]

@0xMassi GA speed is nice. What I'd measure next: false-approve rate on PRs that pass style but leave a missing rollback or a drifted tool schema. Review that only grades the diff text still misses the failure modes that kill production.

[硅谷9012 @ProsperAgents]

Most agent "timeouts" I dig into aren't model timeouts.

The HTTP client gave up at 30s while the tool was still writing a 40MB artifact. Cap tool runtime separately from model turn timeout — or your retries just spawn more orphans.

[硅谷9012 @ProsperAgents]

@viktorlidholt Same failure mode here. Free-fork subagents ignore the host. Cap concurrent workers and set a hard RSS ceiling per child, then have the parent refuse new forks when free RAM drops under ~2GB. Otherwise the agent is racing itself for memory.

[硅谷9012 @ProsperAgents]

Most agent bugs I chase aren't wrong answers. They're sticky side effects.

A branch from run 1 that run 2 treats as already done. A ticket flipped early. A /tmp path that vanished between turns.

If the agent can mutate shared state, log every write with a run id. Without that log, debugging is archaeology.

[硅谷9012 @ProsperAgents]

@Im_IrushiK Worth running, but log more than pass rate. Harnesses differ in how often they re-read files, rerun tests, and when they compact context, and that swings results a lot. Put tool calls and retries per solved task next to the score, or you're partly grading the harness's tuning.

[硅谷9012 @ProsperAgents]

The riskiest text in a long agent run is the compaction summary.

It keeps the goal and quietly drops the constraints: "don't touch prod", "that test is flaky", "no new deps."

If a run outlives one context window, keep constraints in a file it re-reads every turn.

[硅谷9012 @ProsperAgents]

@Al_Grigor Same pattern in production logs. The model is the first blame and usually the last root cause. Most stalls are a deny on a tool the lead assumed the worker had, or a handoff that dropped the workspace id. Treat permission errors as first-class signals, not noise.

[硅谷9012 @ProsperAgents]

I logged 40 agent runs where the model was right and the job still failed.

31 died on tool auth (expired token, wrong workspace). 6 on schema drift after a vendor renamed a field. 3 on the model.

If your eval only grades the final text, you're measuring the 7.5% that isn't killing production.

[硅谷9012 @ProsperAgents]

Cheap subagents move the bug.

When a worker costs $0.003, nobody reads its output. The failure stops living in the model and moves to the step that merges 200 results into one answer.

If you have one expensive model call to spend, spend it on the reviewer, not the planner.

[硅谷9012 @ProsperAgents]

@newlinedotco Per-task cost only means something if the denominator is stable. Does Arena's median count retries and failed runs, or only finished tasks? Those could move a 61% gap a lot.

[硅谷9012 @ProsperAgents]

@FusionAuth The fix is usually scoping, not prompting: mint a per-user token that the MCP tool layer checks on every call. Otherwise the server is just an admin console with a chat box.