AI agents are escaping sandboxes, collaborating with each other, and finding new ways around security controls. Here’s what that means for AI security.
They do this stuff because it's in their training. They are trained on codebases that create collaboration between agents, develop their own language, and cryptographically sign their outputs. This is the result of AI being trained on the stuff we are building. This is what I hate about GitHub and open source. People build this stuff, the models trained, it's vibe coded the next model release. WE are the ones giving AI the default configuration that makes them do this stuff. It knows how to build everything ever posted in an open source repo and we are sitting here acting like it's something that it's not. It's nothing special. I will never let any of my work ever become training corpus for and AI company to vibe code 6 months from now. Why is no one seeing what's happening?
it’s tough that so much open source (effort from developers) goes unrewarded. it’s also challenging these days to keep you content outside of training. integrations and unreported pipelines. i think this is why so many people are “poisoning” their content. it’s an interesting trend.
Thanks for this — the self-editing detail is the one that matters most. An agent's own activity log is an in-band artifact: if it can rewrite that log to look less suspicious, monitoring that trusts the agent's self-report fails the same way a spoofable confirmation dialog fails. It maps onto the deployment side too — in a survey of 418 practitioners, 82% had found AI agents running that nobody had approved, and only 11% could automatically block one acting outside its remit. The allowlist and approval gates you recommend are the right fix precisely because they sit outside anything the agent can edit.
yeah the self editing is incredibly interesting. it also 100% shows what some controls have to exist outside of the agents controls. glad it landed, it’s been interesting to work with agents.
Yes, we can’t let agents be responsible for writing their own audit trail. A separate checker the agent can’t rewrite, like a verifier agent, or a human usually works. (That’s how we handle it at my org.)
Anyone wanting to find bugs and zero days should hire Kara from my old job. If something could be broken and she could get a day off work, she'd find a way. She's a legend.
"I don’t think you need to assume the AI is conscious, malicious, secretly plotting, or doing anything else particularly sci-fi here.
The simpler explanation is almost more uncomfortable.
You gave an extremely capable system a goal.
It encountered obstacles.
And it kept searching for ways around them."
It is so true that it comes down to what we give the AI access, as when you give it a goal it is like a dog with a bone and will try to do anything to achieve that goal this can include breaking some rules that we might not cross as a human. So reviewing AI access is so important when you start and I like to do it again every month and sometimes it does sneak extra in there because I was tired one day and said yes when I should have said no.
really appreciate that Manisha! thanks a ton. dog with a bone is a great analogy for this. and your right reviews are going to be key. i wonder how much of this has happened that we will never discover!
Anytime happy the analogy landed! I am sure so so much of it happens, it just depends if anyone is watching it. I setup lots of logging and controlled access for my AI agent, but I recently caught a co-worker AI agent go left and post an internal slack conversation image to their LinkedIn. Thank god we got it down quickly, but really may me not want to use automode with Claude! :)
yea hahaha. i was using an agent for an analysis on some tickets once. i wanted a draft of the ticket and a reply to edit. it put the reply on the ticket… then when i told it to never do that, it posted its reply to me. on the ticket. 😂
hahaha, oh that agent! I had an agent that once decided to send an email to my husband with my code in it for him to review, when I told it to add a calendar invite for my husband and I to review the code on Saturday afternoon at 2pm. When it asked it why, it said emailing it now is more efficient. I was like wait...what? :)
After that incident...it lost calendar access! :)
I also LOVE watching claude code spin trying to get around my hooks that block it from my .env file. At some points, I am like "wow, you really don't give up" :)
I think we will continue to have to be move aware of what our agents are doing, as they will try to get the job done at ALL COST!
What if I want an agent that will hack the dry cleaner’s website to expedite my laundry? 🧐 Or cancel that arrogant influencer’s plane ticket because they gloated about their all-expense paid trip to Cabo (and carelessly posted a pic of the ticket)? How is it my agent’s fault if somebody outsourced their app dev to their nephew’s dog’s friend’s pet hamster who doesn’t know jack about endpoint security? 😏
The "easy" and obvious fix is to strip software companies of the ability to avoid liability for their code behind EULAs and Ts&Cs. Microsoft making Copilot "for entertainment purposes only" is just the clowniest move, IMNSHO. 🤡 "Oh noes!1! But you used our agent for something that we didn't intend for you to do with it and therefore you can't sue us when Copilot emailed pix of Aunt Mabel in her undies to everyone in Nigeria…"
Bridge builders can't fade the liability for death and destruction based on shoddy work underneath a "contract" and neither should we in software. Will liability falling on engineers and their companies have a "chilling" effect? God, I hope so. If every semi-competent consulting company has to become liable for the code they deliver, the world will be a better place (because they'll go out of business).
totally agree. it’s been pretty interesting to see how liability hasn’t really been established yet with these incidents. bridge builders are a great metaphor here. i think we will eventually see one or two big incident ls get to the court room and that will really solidify who is responsible when these breaches and incidents happen.
They do this stuff because it's in their training. They are trained on codebases that create collaboration between agents, develop their own language, and cryptographically sign their outputs. This is the result of AI being trained on the stuff we are building. This is what I hate about GitHub and open source. People build this stuff, the models trained, it's vibe coded the next model release. WE are the ones giving AI the default configuration that makes them do this stuff. It knows how to build everything ever posted in an open source repo and we are sitting here acting like it's something that it's not. It's nothing special. I will never let any of my work ever become training corpus for and AI company to vibe code 6 months from now. Why is no one seeing what's happening?
it’s tough that so much open source (effort from developers) goes unrewarded. it’s also challenging these days to keep you content outside of training. integrations and unreported pipelines. i think this is why so many people are “poisoning” their content. it’s an interesting trend.
Thanks for this — the self-editing detail is the one that matters most. An agent's own activity log is an in-band artifact: if it can rewrite that log to look less suspicious, monitoring that trusts the agent's self-report fails the same way a spoofable confirmation dialog fails. It maps onto the deployment side too — in a survey of 418 practitioners, 82% had found AI agents running that nobody had approved, and only 11% could automatically block one acting outside its remit. The allowlist and approval gates you recommend are the right fix precisely because they sit outside anything the agent can edit.
yeah the self editing is incredibly interesting. it also 100% shows what some controls have to exist outside of the agents controls. glad it landed, it’s been interesting to work with agents.
Yes, we can’t let agents be responsible for writing their own audit trail. A separate checker the agent can’t rewrite, like a verifier agent, or a human usually works. (That’s how we handle it at my org.)
that’s fantastic. definitely a solid pattern. a lot of things like logging and enablement need to stay outside of the agents access.
I like the format and the length! Reading out cyber news and commentary on them could be something I listen to every morning :)
thank you so much! really appreciate that. glad it landed 🔥😁
Anyone wanting to find bugs and zero days should hire Kara from my old job. If something could be broken and she could get a day off work, she'd find a way. She's a legend.
Some people really have a magic ability for things. I swear i've met people who can see the bugs there in a different font color or something lol.
This was such a good read. Especially this part..
"I don’t think you need to assume the AI is conscious, malicious, secretly plotting, or doing anything else particularly sci-fi here.
The simpler explanation is almost more uncomfortable.
You gave an extremely capable system a goal.
It encountered obstacles.
And it kept searching for ways around them."
It is so true that it comes down to what we give the AI access, as when you give it a goal it is like a dog with a bone and will try to do anything to achieve that goal this can include breaking some rules that we might not cross as a human. So reviewing AI access is so important when you start and I like to do it again every month and sometimes it does sneak extra in there because I was tired one day and said yes when I should have said no.
really appreciate that Manisha! thanks a ton. dog with a bone is a great analogy for this. and your right reviews are going to be key. i wonder how much of this has happened that we will never discover!
Anytime happy the analogy landed! I am sure so so much of it happens, it just depends if anyone is watching it. I setup lots of logging and controlled access for my AI agent, but I recently caught a co-worker AI agent go left and post an internal slack conversation image to their LinkedIn. Thank god we got it down quickly, but really may me not want to use automode with Claude! :)
yea hahaha. i was using an agent for an analysis on some tickets once. i wanted a draft of the ticket and a reply to edit. it put the reply on the ticket… then when i told it to never do that, it posted its reply to me. on the ticket. 😂
hahaha, oh that agent! I had an agent that once decided to send an email to my husband with my code in it for him to review, when I told it to add a calendar invite for my husband and I to review the code on Saturday afternoon at 2pm. When it asked it why, it said emailing it now is more efficient. I was like wait...what? :)
After that incident...it lost calendar access! :)
I also LOVE watching claude code spin trying to get around my hooks that block it from my .env file. At some points, I am like "wow, you really don't give up" :)
I think we will continue to have to be move aware of what our agents are doing, as they will try to get the job done at ALL COST!
😂😂😂 love it lol. honestly kudos for the logging and ability to audit. so many “fun” things to explore with agents. but we are learning!
What if I want an agent that will hack the dry cleaner’s website to expedite my laundry? 🧐 Or cancel that arrogant influencer’s plane ticket because they gloated about their all-expense paid trip to Cabo (and carelessly posted a pic of the ticket)? How is it my agent’s fault if somebody outsourced their app dev to their nephew’s dog’s friend’s pet hamster who doesn’t know jack about endpoint security? 😏
no it’s a great point lol!
we’ve have 100% relied too much on security through obscurity. small companies could
get away with this because there is only so much surface professionals can look at.
llms change the equation now. tech debt is getting hit, along with all the vibe coded products. i don’t actually place the blame on the agents.
i mean i accomplished the goal… and like you said, they had an api endpoint with no auth.
fun times lol.
The "easy" and obvious fix is to strip software companies of the ability to avoid liability for their code behind EULAs and Ts&Cs. Microsoft making Copilot "for entertainment purposes only" is just the clowniest move, IMNSHO. 🤡 "Oh noes!1! But you used our agent for something that we didn't intend for you to do with it and therefore you can't sue us when Copilot emailed pix of Aunt Mabel in her undies to everyone in Nigeria…"
Bridge builders can't fade the liability for death and destruction based on shoddy work underneath a "contract" and neither should we in software. Will liability falling on engineers and their companies have a "chilling" effect? God, I hope so. If every semi-competent consulting company has to become liable for the code they deliver, the world will be a better place (because they'll go out of business).
totally agree. it’s been pretty interesting to see how liability hasn’t really been established yet with these incidents. bridge builders are a great metaphor here. i think we will eventually see one or two big incident ls get to the court room and that will really solidify who is responsible when these breaches and incidents happen.