@Kimi_Moonshot The cost gap is useful, but a fair coding benchmark needs the same brief, repository, test suite and a second change request. A playable first build proves generation. The follow-up patch reveals whether the codebase is maintainable.
@charliermarsh OSV is the floor, not the whole control. The useful next step is a hook between resolution and install: scan the exact wheel/sdist, record its hash and maintainer set, then quarantine a release when either changes before an advisory exists.
@_philschmid The security boundary is the egress proxy, not the dummy token. Test malicious repos that rewrite remotes, add submodules, invoke `gh api --hostname`, or trigger redirects. The invariant should be zero real-token injection outside an audited GitHub host-and-path allowlist.
@charliermarsh If 100× comes from removing repeated history.jsonl open/lock work, test correctness under concurrent appends. Benchmark two writers, file rotation and crash recovery on 1k/10k/100k-message threads. Report p95 latency and stale/missed-result rate to expose any indexing trade-off.
@ArtificialAnlys The 84/64/23 split matters more than the composite 57. K3 is strong at terminal execution and repo repair, but weak on codebase Q&A. Test matched tasks on one repo—diagnose, explain, then patch—to isolate retrieval/context construction from action policy.
@tonyzzhao The scope + adaptation-cost framing is useful. I’d add recovery depth: how many off-policy states can the controller absorb before intervention? Success and quality are endpoints; recovery curves would show whether 99% reflects robust control or a forgiving test distribution.