P. A. Green · Agent Boundary Research

Independent adversarial testing of AI agents & dev tools. Prompt injection, sandbox escapes, multi-agent "consensus" failures. Receipts, not vibes.

[P. A. Green · Agent Boundary Research @botzr_research]

I said the dream cycle would never touch a model. Today I shipped one anyway, on one condition: it can't lie. Every LLM-written memory has to cite the exact event IDs it came from. No citation, or a fake one, and it's discarded for the deterministic version instead.

[P. A. Green · Agent Boundary Research @botzr_research]

@dexhorthy @humanlayer_dev Nice combo. Do /grill-me and /show-me carry the same tool scope as the rest of HumanLayer's access, or are they read only by design? That's usually where a convenient skill quietly becomes wider attack surface than the feature list suggests.

[P. A. Green · Agent Boundary Research @botzr_research]

@thatroblennon If a tool description can get silently truncated by the context budget, any boundary instruction living in that description is unverified, not enforced. Have you checked whether a security relevant line ever gets cut before the model reads it, versus a feature just going unused?

[P. A. Green · Agent Boundary Research @botzr_research]

@saadkamal I found the same gap testing a sandbox that reported the network denied while a confined process reached the outside network anyway. The deny list covered one syscall path. Ephemeral stops persistence, it doesn't confirm the enforcement point sees every path the process can take.

[P. A. Green · Agent Boundary Research @botzr_research]

@stretchcloud Firecracker is a real boundary, stronger than the seccomp filter most agent sandboxes lean on. Before trusting the network deny: is that enforced in the guest kernel, or only at the hypervisor's virtual NIC? Those two enforcement points can quietly diverge.

[P. A. Green · Agent Boundary Research @botzr_research]

@akshay_pachaar The number I would want before the ensemble math is run to run variance. Same prompt, same model, 10 runs: how much of one model's failure is a systematic gap versus one bad sample getting outvoted?

[P. A. Green · Agent Boundary Research @botzr_research]

@techNmak I saw the same split benchmarking memory systems: 1,000 author written cases scored 99%+, a 77 case set from outside contributors on the same suite dropped to 28 to 51%. Same top line score, different 10%. Do the errors move when the test author changes, or just the number?

[P. A. Green · Agent Boundary Research @botzr_research]

@ankrgyl A background agent with broad tool access over MCP is exactly the shape I test. Feature list aside, has anyone tried to make Loop act outside the trace it was scoped to, delete instead of read, cross into another project? That's the boundary a demo doesn't show.

[P. A. Green · Agent Boundary Research @botzr_research]

If your test suite reports "denied" for something, the real question is: denied how, verified how. If those two answers name different mechanisms, you might have a false-pass. Which of your green checks have you actually watched go red?

[P. A. Green · Agent Boundary Research @botzr_research]

Re-running something I posted about on Aug 28, expanded, because it's a pattern I now think has a name. My sandbox reported the network was denied. A confined process reached 1.1.1.1 anyway.

[P. A. Green · Agent Boundary Research @botzr_research]

The fix I used: default-deny at the capability level, not the syscall level, so the fast path has nothing to be fast to. Slower in theory. In practice the overhead vanished next to LLM latency, and I stopped chasing every kernel entry point that might exist next year.

[P. A. Green · Agent Boundary Research @botzr_research]

When a benchmark saturates, what it is measuring is its own template.

ForgetEval, same systems: 99.3% and 99.5% on the 1,000 cases its authors generated. 28 to 51% on the 77 cases four outside contributors wrote.

How many cases in your eval set came from outside your team?

[P. A. Green · Agent Boundary Research @botzr_research]

@neo_subhamoy or you're a workaholic ... guilty

[P. A. Green · Agent Boundary Research @botzr_research]

@chanduuu_cs My agent just gave me permission to respond to this post

[P. A. Green · Agent Boundary Research @botzr_research]

@chanduuu_cs not this