@zhangliang0115 Right. Tests only cover the paths they run. The grep also finds dead imports, config keys and docs that still name the old helper. I run it on the whole repo, not just src.
Agents treat existing code as the style guide, so one bandaid becomes the template.
Better: ask for the correct fix, not the smallest. Put "fix root causes, no new workarounds" in CLAUDE.md/AGENTS.md. Review diffs for patterns they introduce, not just green tests.
quick fixes considered harmful: you might be making your agents write worse code when you ask for the smallest or quickest possible change
what seems like a harmless bandaid may lead to that pattern being copied by the next agent. before you know it that pattern has spread everywhere
we used to write bandaid fixes by hand because we rarely had the time to do the right thing. agents change this calculus. sometimes you want it to fix the root cause instead even if it’s more invasive
@OrbitalFund Same. I keep the test and lint commands at the top of AGENTS.md and ask the agent to paste the passing output before it says done. A "done" with no output goes back.
The third question is the one most harnesses skip. "Done" should be a command, not a feeling.
Put the exact test + lint command in AGENTS.md / CLAUDE.md. Rule: the agent may say done only after it runs it and pastes the passing output.
Same model, checkable "done".
Claude Code vs Codex isn't the real question.
The real question is your harness:
→ What context does the agent see?
→ What tools can it call?
→ How does it know it's done?
Same model, different harness, completely different results.
What's in your harness that most people skip?
@AtlasCapAi Strong addition. Give each side-effecting call a request id, have the tool return the existing result on a repeat, and make the eval count actions taken, not just the final message. Two charges is a fail even if the reply says success.
pass@k flatters customer-facing agents. Users don't get k tries. They need it right every time.
Anthropic's agent evals guide: at 75% per-trial success, pass^3 (all 3 trials pass) is about 42%.
Gate launches on pass^k, not pass@1.
https://t.co/TBhm3Pv3Z5
@NicoHao4s Good plan. Keep the pre-merge check small: run only the cases tagged for the area the change touches, so it stays fast enough that nobody skips it. The full pack can stay nightly.
Own the verification, not the typing.
Agents make code cheap. Proof that it works is still scarce. Write the spec, the tests and evals that catch the agent being wrong, and the review rules that stop bad patterns from spreading. That is the senior skill now.
@ARLPROTOCOL Agreed. A quick test: write the steps down before you build. If the list holds and only the inputs change, ship a chain. If the next step keeps depending on what the last tool call returned, that part earns an agent. Most good systems are a chain with one agentic step.
Locks and "don't touch file X" prompts are the wrong fix. Give each agent its own checkout:
git worktree add ../agent-b -b agent-b
(Claude Code: claude --worktree agent-b)
Split tasks by file ownership. Each agent rebases and runs tests before merge.
Agentic AI interview question:
Two agents are editing the same codebase at the same time.
One just overwrote the other’s changes.
How do you let them work in parallel?
Public scores won't tell you how a tool-selection model does on your tools.
Log ~100 real agent turns, each labeled with the tool that should have been picked. Run every candidate on that set with your exact tool names and descriptions. Pick on that score, not the leaderboard.
Open-source multimodal decision model, built on Gemma 4 31B!
Karpo Decision 31B is designed for fast, accurate decision-making in AI agents, including tool selection, browser navigation, retrieval, ranking, and safety checks.
The model uses a dedicated readout head to score possible answers directly and return calibrated probabilities.
One of the interesting parts is how it handles multiple decisions.
Karpo uses branch attention to process shared context once, while keeping each question independent.
For example, a browser agent can examine a webpage and make several decisions about which element to click, whether to call a tool, or whether to ask for clarification, all within a single forward pass.
It also supports native multimodal input, so these decisions can be made using both text and images.
Key capabilities:
- Answer multiple independent questions in one inference pass
- Support yes/no, multiple-choice, and scoring decisions
- Return calibrated probabilities for confidence-based actions
- Handle up to 256K context tokens
- Optimized inference using FP8, CUDA graphs, and custom attention kernels
- Six open-weight model sizes from 0.8B to 31B
Karpo Decision 31B ranks #1 among 300 models on Decision Index, scoring 73.9 on its Full evaluation.
It also achieves 95ms median latency on a single RTX PRO 6000 across the public benchmark suite.
The team focused on generalization as well, using unseen-domain evaluations and checkpoint fusion to improve performance beyond familiar benchmarks.
Both the model weights and inference code are open source.
I've shared the model in the comments!
4 Grok Bot routines run 5 times a day for me. I set each up in one sentence.
8 AM + 8 PM: meds reminder. 2:30 AM: safe disk cleanup. 10 AM: sites-up and action-only inbox check. 11:50 PM: memory backup to GitHub.
Which would you set up first?
@NicoHao4s That tag pays off fast. After a few weeks, sort the cases by tag. If one area, like the prompt template or a tool schema, keeps showing up, add a review step or a cheap pre-merge check there, not just more cases.
@zhangliang0115 Yes. One check that helps: after the change, grep for the old helper's name. If anything still calls it, the change isn't done. Then delete the old one in the same change, so nothing new can start using it.
@Wv3EcACHtizRjZK Agreed. I'd show it as a per-task table: run each task k times. Tasks that never pass are a capability or spec gap. Tasks that pass only some runs are variance, usually a vague prompt or noisy tool output. Different fixes, and the aggregate hides which one you have.
@zhangliang0115 "Clean start" is the one reason I never accept. If the old helper is truly wrong, the fix is to refactor it and move its callers in the same change, not to add a second copy next to it.
@XLNC_CO@shaharfrish Agreed on ties. Clustered flips are a useful signal. If they bunch at one score level, I'd check that level's rubric wording first, since vague anchors are a common cause. Did tightening the anchors fix the harsh/lenient swing?
Treat the judge prompt as a model under test. Keep a frozen set of human-labeled items. On every prompt edit, rerun every judge on that set and check agreement with the human labels, not raw scores. Ship the edit only if agreement holds. Version prompt and model together.
How do you keep an AI judge honest between prompt versions?
Three AI judges scored one set of ideas 55-84. After one prompt edit, their scores on that set fell to 3-22%.
@NicoHao4s The promote-back loop is the key part. Also tag each promoted case with the change that broke it. After a few weeks you can see which kinds of edits cause regressions, and that tells you where to add cases.
@zhangliang0115 Nice gate. One more: have it name the closest existing helper and say in one line why it doesn't fit. That makes it easy to tell "didn't search" from "searched but didn't like the old code".
An Optimizely flaw let an attacker inject code into A/B-test scripts on 178 orgs' sites, incl. Zoom, Salesforce and HSBC. It came from a trusted domain with a valid cert.
Every third-party script runs with your site's power. Keep few, use CSP, watch them for changes.