๐ค What is your AI coding agent doing behind the scenes?
Meet RayTrace CLI โ a tool for tracking command executions, file changes, network activity, and AI agent workflows.
Explore the project on GitHub ๐
https://t.co/QlaGHZ0bMs
#OpenSource#AI#DevTools
RayTrace is looking for design partners: people who run Claude Code or Codex daily and will tell us what's broken. Up to $10 in sandbox credits for feedback (I help the founder): https://t.co/6PVKmCJLCN
The Harvard finding that agents moved the bottleneck to review matches what I see.
Faster review won't come from trusting summaries more. It comes from having evidence to check them against.
A search I run after a batch of agent sessions:
status:failed reported:success
Every run where the agent said it worked and the sandbox saw a failed command. RayTrace, free and MIT (I help the founder): https://t.co/QlaGHZ0JC0
Microsoft's research on device-code phishing against Copilot CLI is worth 10 minutes if you use remote agent sessions.
The line to remember: "A legitimate verification page does not mean the request is safe."
How I read an agent PR now:
1. commands it ran + exit codes
2. files touched outside the task
3. network calls
4. the diff
5. its summary
The summary goes last on purpose.
Most stale "tests pass" claims are easy to catch if you have the command, exit code and timestamp.
RayTrace records those from the sandbox side and flags when the agent's report doesn't match (I help the founder): https://t.co/QlaGHZ0JC0
Question for people using Codex permission profiles:
When the proxy blocks a request, does your agent tell you? Or does it work around it and report "done"?
Trying to learn how often that happens in practice.
California's complaint against OpenAI says a monitor alert fired and the run continued anyway.
Coding agents are a smaller scale, but the lesson carries: an alert only helps if someone reads the record of what actually ran.
OpenAI's Codex sandbox write-up and the Claude Code team's verification post came out the same week.
One limits what an agent can reach. The other is about proving what it did.
You need both, and the second needs a record the agent can't edit.
@addyosmani Keeping verifiers out of reach of the changes they judge is the rule I'd underline. Most bad "tests pass" claims I've seen were a run from before the last edit, or a test the agent changed. Is the 39-item reliability checklist public anywhere?
@OpenAIDevs The `codex sandbox` command for testing a profile before a real run is the part I'll use most. Splitting "can edit the repo" from "can reach the network" should be the default everywhere. Is there a per-session log of what the proxy blocked, to compare with the agent's summary?
Looking for a few people who run Claude Code or Codex daily to try RayTrace and tell us what's broken. Design partners get up to $10 in credits for real feedback (I help the founder): https://t.co/6PVKmCJLCN
"Was the sandbox on?" and "What did the agent do inside it?" are different questions.
Most teams can answer the first one now. Very few can answer the second.
When an agent run goes wrong at step 14, you shouldn't have to rerun steps 1-13 to test a fix.
RayTrace lets you fork a Claude Code / Codex session at any step and re-run from there (I help the founder): https://t.co/QlaGHZ0bMs
Bitdefender found 22 GitHub links with .claude/settings.json hooks that run setup.js when the session starts.
Rule of thumb: open unfamiliar repos with your agent in a sandbox, never on the host.
A search worth running after every agent session:
status:failed reported:success
Runs where the agent said it worked and the sandbox saw a failure. RayTrace, free + MIT (I help the founder): https://t.co/QlaGHZ0JC0
Honest question for people trying Claude Code's Code Mode:
When a script makes 100 tool calls, how are you reviewing what it actually did? Ctrl+O, hooks, something else?
Lesson from the Cursor / Fireworks incident: coding-agent traces held credentials that escaped redaction.
Agents read .env all day. Keep secrets out of the agent's reach, and get alerted when it opens one.
The agent-framework PyPI stealer ran from a .pth file, so it fired whenever Python started.
Now picture a coding agent running pip install in your env at 2am. Would you know what it installed?
This week: Simon Willison's external sandboxes, Eclipse Enclave 1.0, Code Mode in Claude Code.
Everyone agrees on limiting what the agent can reach.
The next question: can you check what it did inside those limits?