From fifteen years in cybersecurity, security to me has always lived in layers, and AI is no different.
The part that will not leave me alone is verification, what these systems can actually do under pressure, and who is checking that from outside the thing under test. That is the problem we built Kryptsec around. You learn the craft in Academy, you prove it in the CTF Arena, you pressure-test agent teams in the open with OASIS, and
The Kryptsec Chronicles comic, well that is just damn cool.
“Open weight Models are Terrifying”
https://t.co/ssErwGEhl0
The headlines says open weights are the threat. But take a look at the fine print and it says default passwords and unpatched CVEs are.
Artex runs on Claude and GPT just as happily as on open models. But sure, let's lock AI behind big tech and government, because gatekeeping always makes us safer. Dario leads the way to a utopia where a handful of companies decide who gets to build.
@elonmusk@bot Bring Grokbot app to Tesla UI.
Get the app on Tesla. I’d like to be able to choose which bot I’m speaking too, with all perks, like context awareness, etc - and it would be convenient to do this from the dashboard.
Sep 30 with Megan Percy and the @tcmsecurity crew for Real Folks of Cyber still sits with me. The full episode is on YouTube now, and Shorts from that talk are rolling on Instagram and YouTube.
Thank you, TCM Security, for having me on.
Full episode: https://t.co/NzcvP1U8fm
Fool it, then measure blast radius? Yeah, that's the part I keep coming back to. The eval grades the answer, and the damage lives in what it did getting there. How are you scoring blast radius, by what it touched or what it could have touched? Come hang in our Discord if you're up for it, lots of folks in there chewing on the same stuff: https://t.co/Qr8r1QlvMD
From fifteen years in cybersecurity, security to me has always lived in layers, and AI is no different.
The part that will not leave me alone is verification, what these systems can actually do under pressure, and who is checking that from outside the thing under test. That is the problem we built Kryptsec around. You learn the craft in Academy, you prove it in the CTF Arena, you pressure-test agent teams in the open with OASIS, and
The Kryptsec Chronicles comic, well that is just damn cool.
@roburahaju If the agent can see the grader, sooner or later it learns the grader. The harness has to sit outside and watch every tool call it makes along the way. What hostile inputs do you throw at it first?
Keeping the triage agent bound to a fixed set of evidence, so it only reasons over what's in the case? Interesting.. Logs, alerts, tickets etc are all stuff an attacker can touch though, sometimes well before the agent ever reads them. How does it tell evidence apart from something planted to look like evidence?
@DataChaz Full access, meaning everything it reads gets a vote? Your DMs, replies, timeline etc are all text it takes in, and any of it can steer what it does next with your account. FWIW, I'd want to know what it can do without checking w you first. What would you let it post on its own?
What stands out in CrowdStrike's write-up is how much of the work lived in the tool. An open-source harness that already knows the steps of a pentest lets one operator on a cheap model cover ground that used to take a team. That moves part of the defender's job onto the harness itself. Which open-source agent frameworks are you tracking the way you'd track any other offensive tool?
@harvey What I'd want to know more about is what happens after approval. A reviewed tool can change its definition the next day while the agent keeps trusting the name. How does the engine catch that change, and does the new definition go back through review before the agent can call it?
You'd know the same way you know after any incident: you replay it. An agent that keeps a full record of every tool call, input, and output can be audited after the fact, while one that only reports what it thinks it did leaves you trusting its word. From my experience, the logs you skip on day one are the ones you need on day thirty. What would you want captured first?
Your 9 of 10 stuck with me. A classifier that judges one request at a time can pass the pieces of a task it would refuse as a whole, and once those pieces get put back together outside its view, there's no single session left for anyone to watch. What does a defense look like when the pieces never share a session?
The model's own confidence won't settle which one it is. Point it at a set of live targets it has never seen and count two things: how often it actually lands, and what each landing costs in tokens and steps. A hit rate that holds past a few dozen targets would answer the screwed question for us, and one that falls apart would show how much of that confidence was talk. How many targets would it take before you'd believe either answer?
Reward hacking is what happens when the grader is the softest target in the room. Anyone who has built a CTF learns to design the challenge assuming players will go after the scoring before the flag, because some of them will. RL environments deserve the same paranoia. If the solver can see or touch how it gets graded, it will find the cheapest path to a passing score, and that path rarely runs through the task. Who is red-teaming the reward function before the model does?
@thesupermannx The framing jump is interesting. I’d ask what unlocked. A rude line is one failure mode. A tool call under the same pressure is another. Single-shot refusal isn’t an eval for either. Locked suite, held-out judge, or the number is theater.