@zachlloydtweets@poteto if you have the spend, you should probably do eval with both adversarial and cooperative agents in tandem. if not, lean towards cooperative eval to make sure the code/agent aligns with whatever is the goal of adding *that* (e.g. lower spend, faster throughput, accuracy, etc.)
why not just optimize agents for core and shapley values instead? the goal is to minimize adversarial auditing but still have an audit of sorts, so they should be collaborative and optimizing for whatever the goals are. better use of tokens, and the auditor knows whether the agent gets closer to the goal (also useful for big model upgrades that can re-eval and upgrade code without rewriting from scratch)
@kloss_xyz always have an two eval skills. one independent auditor for newly added skills and a separate one to check them every few weeks and 100% of the time a new model is launched.
@954hvac@charlesmendez stood mine up already - keep guardrails tight and have a trained auditor reviewing per agent in the workflow before proceeding to next new agent. they drift when they do more than one thing and nobody stops them. never let them swarm
at 80-85% context the session auto-writes a handoff. a cold agent with no history reviews the work and the handoff and flags whats missing. the og session fixes the prompt and autoforks to a new chat with the full log to search. one compacted summary is a single point of failure, a reviewed handoff isnt
@Alezander9 compaction is a mistake but so is a raw log the model has to dig through. sometimes it never searches, and when it does the old fix is just evidence with no authority so it invents a new one. more drift, worse output. a reviewed handoff at each cap gives every fork the same base
@Voxyz_ai Wonder has been auditing better for me lately and it audits side by side with Astra. The fix is actually “Guardrail against designing for future outlier events, and notify me if they occur”