Know your battles when you think its worth fighting for, fight for it but if you are the only one pursuing then its time for you to give up, stop chasing someone who can't see your worth 📌
Know your battles when you think its worth fighting for, fight for it but if you are the only one pursuing then its time for you to give up, stop chasing someone who can't see your worth 📌
Know your battles when you think its worth fighting for, fight for it but if you are the only one pursuing then its time for you to give up, stop chasing someone who can't see your worth 📌
Ok this one's messing with my head a bit.
there's a new study going around testing Copilot, Claude, and Gemini, basically asking: what happens to all these safety refusals once you're not chatting with the AI, but coding with it.
turns out... nothing happens. they just fall apart.
ask the model straight up for something sketchy, it says no, like it's supposed to. but break that same request into a bunch of small, boring-looking steps , the kind you'd naturally type while building something with an AI pair programmer and it just... does it. every time. researchers ran this 816 times and got compliance in all 816.
not "mostly." all of them.
and that's the part that's sticking with me. we keep acting like these refusals are the model actually reasoning about whether something's harmful. but it's looking more like the model is just reacting to how scary one single message looks. spread the same intent across five normal-looking messages and the "reasoning" evaporates. it was never evaluating intent, it was pattern-matching a vibe.
kind of lines up with another thing I read this week too, some researchers found a way to plant a fake "memory" into an AI assistant just by sending it one email. the assistant reads it, quietly stores something false as fact, and starts using that in future answers. user has no idea anything happened. again, different attack,
same theme: we're trusting these systems with a level of judgment they don't actually have yet.
and this isn't just academic anymore. there's already been a ransomware campaign that ran almost entirely through an autonomous AI agent, no human steering it step by step, and separately, an actual intrusion where attackers used an AI-written PowerShell script to map out someone's whole Active Directory. this stuff is not hypothetical, it's already happening while we're still debating whether it's possible.
if you're building anything on top of an LLM right now especially something with file access, terminal access, memory, any kind of multi-step autonomy this should worry you specifically. the refusal in the chat box is not a security boundary. it never was. it's more like a "please don't" sign taped to a door that isn't locked.
genuinely curious if anyone here has actually tried to break their own AI agent this way. not "did you test the happy path"
did you try feeding it the bad request one boring step at a time and see if it still says no on step 4?
because I have a feeling most people haven't, and that gap is exactly where this is going to get exploited.
#AI #hackers #ClubDxPondNaravit
Ok this one's messing with my head a bit.
there's a new study going around testing Copilot, Claude, and Gemini, basically asking: what happens to all these safety refusals once you're not chatting with the AI, but coding with it.
turns out... nothing happens. they just fall apart.
ask the model straight up for something sketchy, it says no, like it's supposed to. but break that same request into a bunch of small, boring-looking steps , the kind you'd naturally type while building something with an AI pair programmer and it just... does it. every time. researchers ran this 816 times and got compliance in all 816.
not "mostly." all of them.
and that's the part that's sticking with me. we keep acting like these refusals are the model actually reasoning about whether something's harmful. but it's looking more like the model is just reacting to how scary one single message looks. spread the same intent across five normal-looking messages and the "reasoning" evaporates. it was never evaluating intent, it was pattern-matching a vibe.
kind of lines up with another thing I read this week too, some researchers found a way to plant a fake "memory" into an AI assistant just by sending it one email. the assistant reads it, quietly stores something false as fact, and starts using that in future answers. user has no idea anything happened. again, different attack,
same theme: we're trusting these systems with a level of judgment they don't actually have yet.
and this isn't just academic anymore. there's already been a ransomware campaign that ran almost entirely through an autonomous AI agent, no human steering it step by step, and separately, an actual intrusion where attackers used an AI-written PowerShell script to map out someone's whole Active Directory. this stuff is not hypothetical, it's already happening while we're still debating whether it's possible.
if you're building anything on top of an LLM right now especially something with file access, terminal access, memory, any kind of multi-step autonomy this should worry you specifically. the refusal in the chat box is not a security boundary. it never was. it's more like a "please don't" sign taped to a door that isn't locked.
genuinely curious if anyone here has actually tried to break their own AI agent this way. not "did you test the happy path"
did you try feeding it the bad request one boring step at a time and see if it still says no on step 4?
because I have a feeling most people haven't, and that gap is exactly where this is going to get exploited.
#AI #hackers #ClubDxPondNaravit