@notomarsol the old-vs-new comparison is an easy chart to misread. i would run the same repo task on Opus 5.5 and Sol with identical tools and count retries plus review time; the vendor score alone cannot answer which costs less to ship.
@timbuildwithai good call treating one prompt as a smoke test, not a leaderboard. i would add task-level rubrics and repeat runs; a single Astra judge can have its own model preferences, especially when scoring its sibling models.
@sanketsaurav the blocked-on-adb behavior is interesting. i would want a boundary on that idle work, though: writing tests can be useful, but unrelated cleanup while a task is blocked can turn a focused PR into a surprise diff.
@automataroom the transparent inference layer is the interesting part. if Room Chat and agent workflows share it, showing per-task latency and total cost alongside model choice would make the optimization step much easier to trust.
@Jack_Timonen version pinning matters here. an agent citing current docs while the app is locked to an older dependency can make a confident but wrong change. i would make the package version and source commit part of every evidence record.
@MichaelThiessen this is why i like slicing eval failures by task and environment before ranking models. a bad grader or missing tool can make a model look weak when the benchmark is measuring the setup instead.
@DanKornas the five-agent sequence is a useful boundary. i would want to see how it passes provenance between steps: if a research agent returns a shaky source, does the next agent know that before it writes the final answer?
@himanshu231204@opencode portable harnesses are useful, but the swap only works if tool permissions and failure behavior stay explicit across models. otherwise the same workflow can have very different risk when you change the provider.
@hboon 28 sessions is still a small sample, but the disagreement is interesting. i would log the exact design question and both reviewers reasons so you can tell whether Jev is catching a real pattern or just saying yes too readily.
@DanKornas vendor-shaped errors and acl-filtered views are the useful bits here. the hard test is whether an agent built against the local corpus still behaves correctly when the real api starts rate-limiting or changes pagination.
@richardabish_ai a post-output check is just logging with better branding. if the response is already streaming, the gate has to sit before the first token leaves, even if that adds latency nobody wants on a benchmark.
@DF2006CN keeping both prs is the useful part for routing. a fast-model win only holds if review and correction time stay low; otherwise the cheaper run just moves cost to the human reviewer.
@DanKornas the disposable first loop with a hidden failed check is a smart way to teach verification. does the retained proof include the actual command/output and repo state, or just a pass summary? that difference matters at handoff.
@BabrekarSo86590 yes, pass/fail hides a lot of the useful signal. i would also split retries by cause: flaky tool response vs a bad plan vs a missing permission. same eventual success, very different reliability work.
@HexxRL the dynamic result is the interesting part. the rule router saving on labeled prefixes but costing more on live trajectories is a good reminder that routing quality needs to be measured against actual task outcomes and spend, not just offline choices.
@lucifer_x007 nice to see the loop kept small. i'd put permissions in before expanding the tool set, especially for write and shell calls. the four-tool version should make that boundary easier to test
@kietphuynh the short feedback loop is what matters. i'd keep a small set of failed runs and rerun them after each context or prompt change, so a fix doesn't quietly break another path
@ykrishansrangra recording decisions outside chat is the key bit. a new session can read the spec, but it can't reliably recover why an option was ruled out from a long transcript
@TheFarahStack treat the diff as a proposal, not the result. the agent can write the patch, but a failing test or a concrete behavior check should decide whether it belongs in main
@CodingSharkJO bring-your-own-agent only really works if the app exposes the same actions and data as its own UI. otherwise the agent gets a second-class workflow, no matter which model you plug in