AI agents are escaping sandboxes, collaborating with each other, and finding new ways around security controls. Here’s what that means for AI security.
They do this stuff because it's in their training. They are trained on codebases that create collaboration between agents, develop their own language, and cryptographically sign their outputs. This is the result of AI being trained on the stuff we are building. This is what I hate about GitHub and open source. People build this stuff, the models trained, it's vibe coded the next model release. WE are the ones giving AI the default configuration that makes them do this stuff. It knows how to build everything ever posted in an open source repo and we are sitting here acting like it's something that it's not. It's nothing special. I will never let any of my work ever become training corpus for and AI company to vibe code 6 months from now. Why is no one seeing what's happening?
it’s tough that so much open source (effort from developers) goes unrewarded. it’s also challenging these days to keep you content outside of training. integrations and unreported pipelines. i think this is why so many people are “poisoning” their content. it’s an interesting trend.
"I don’t think you need to assume the AI is conscious, malicious, secretly plotting, or doing anything else particularly sci-fi here.
The simpler explanation is almost more uncomfortable.
You gave an extremely capable system a goal.
It encountered obstacles.
And it kept searching for ways around them."
It is so true that it comes down to what we give the AI access, as when you give it a goal it is like a dog with a bone and will try to do anything to achieve that goal this can include breaking some rules that we might not cross as a human. So reviewing AI access is so important when you start and I like to do it again every month and sometimes it does sneak extra in there because I was tired one day and said yes when I should have said no.
really appreciate that Manisha! thanks a ton. dog with a bone is a great analogy for this. and your right reviews are going to be key. i wonder how much of this has happened that we will never discover!
Anytime happy the analogy landed! I am sure so so much of it happens, it just depends if anyone is watching it. I setup lots of logging and controlled access for my AI agent, but I recently caught a co-worker AI agent go left and post an internal slack conversation image to their LinkedIn. Thank god we got it down quickly, but really may me not want to use automode with Claude! :)
yea hahaha. i was using an agent for an analysis on some tickets once. i wanted a draft of the ticket and a reply to edit. it put the reply on the ticket… then when i told it to never do that, it posted its reply to me. on the ticket. 😂
hahaha, oh that agent! I had an agent that once decided to send an email to my husband with my code in it for him to review, when I told it to add a calendar invite for my husband and I to review the code on Saturday afternoon at 2pm. When it asked it why, it said emailing it now is more efficient. I was like wait...what? :)
After that incident...it lost calendar access! :)
I also LOVE watching claude code spin trying to get around my hooks that block it from my .env file. At some points, I am like "wow, you really don't give up" :)
I think we will continue to have to be move aware of what our agents are doing, as they will try to get the job done at ALL COST!
What if I want an agent that will hack the dry cleaner’s website to expedite my laundry? 🧐 Or cancel that arrogant influencer’s plane ticket because they gloated about their all-expense paid trip to Cabo (and carelessly posted a pic of the ticket)? How is it my agent’s fault if somebody outsourced their app dev to their nephew’s dog’s friend’s pet hamster who doesn’t know jack about endpoint security? 😏
The "easy" and obvious fix is to strip software companies of the ability to avoid liability for their code behind EULAs and Ts&Cs. Microsoft making Copilot "for entertainment purposes only" is just the clowniest move, IMNSHO. 🤡 "Oh noes!1! But you used our agent for something that we didn't intend for you to do with it and therefore you can't sue us when Copilot emailed pix of Aunt Mabel in her undies to everyone in Nigeria…"
Bridge builders can't fade the liability for death and destruction based on shoddy work underneath a "contract" and neither should we in software. Will liability falling on engineers and their companies have a "chilling" effect? God, I hope so. If every semi-competent consulting company has to become liable for the code they deliver, the world will be a better place (because they'll go out of business).
totally agree. it’s been pretty interesting to see how liability hasn’t really been established yet with these incidents. bridge builders are a great metaphor here. i think we will eventually see one or two big incident ls get to the court room and that will really solidify who is responsible when these breaches and incidents happen.
They do this stuff because it's in their training. They are trained on codebases that create collaboration between agents, develop their own language, and cryptographically sign their outputs. This is the result of AI being trained on the stuff we are building. This is what I hate about GitHub and open source. People build this stuff, the models trained, it's vibe coded the next model release. WE are the ones giving AI the default configuration that makes them do this stuff. It knows how to build everything ever posted in an open source repo and we are sitting here acting like it's something that it's not. It's nothing special. I will never let any of my work ever become training corpus for and AI company to vibe code 6 months from now. Why is no one seeing what's happening?
it’s tough that so much open source (effort from developers) goes unrewarded. it’s also challenging these days to keep you content outside of training. integrations and unreported pipelines. i think this is why so many people are “poisoning” their content. it’s an interesting trend.
This was such a good read. Especially this part..
"I don’t think you need to assume the AI is conscious, malicious, secretly plotting, or doing anything else particularly sci-fi here.
The simpler explanation is almost more uncomfortable.
You gave an extremely capable system a goal.
It encountered obstacles.
And it kept searching for ways around them."
It is so true that it comes down to what we give the AI access, as when you give it a goal it is like a dog with a bone and will try to do anything to achieve that goal this can include breaking some rules that we might not cross as a human. So reviewing AI access is so important when you start and I like to do it again every month and sometimes it does sneak extra in there because I was tired one day and said yes when I should have said no.
really appreciate that Manisha! thanks a ton. dog with a bone is a great analogy for this. and your right reviews are going to be key. i wonder how much of this has happened that we will never discover!
Anytime happy the analogy landed! I am sure so so much of it happens, it just depends if anyone is watching it. I setup lots of logging and controlled access for my AI agent, but I recently caught a co-worker AI agent go left and post an internal slack conversation image to their LinkedIn. Thank god we got it down quickly, but really may me not want to use automode with Claude! :)
yea hahaha. i was using an agent for an analysis on some tickets once. i wanted a draft of the ticket and a reply to edit. it put the reply on the ticket… then when i told it to never do that, it posted its reply to me. on the ticket. 😂
hahaha, oh that agent! I had an agent that once decided to send an email to my husband with my code in it for him to review, when I told it to add a calendar invite for my husband and I to review the code on Saturday afternoon at 2pm. When it asked it why, it said emailing it now is more efficient. I was like wait...what? :)
After that incident...it lost calendar access! :)
I also LOVE watching claude code spin trying to get around my hooks that block it from my .env file. At some points, I am like "wow, you really don't give up" :)
I think we will continue to have to be move aware of what our agents are doing, as they will try to get the job done at ALL COST!
😂😂😂 love it lol. honestly kudos for the logging and ability to audit. so many “fun” things to explore with agents. but we are learning!
What if I want an agent that will hack the dry cleaner’s website to expedite my laundry? 🧐 Or cancel that arrogant influencer’s plane ticket because they gloated about their all-expense paid trip to Cabo (and carelessly posted a pic of the ticket)? How is it my agent’s fault if somebody outsourced their app dev to their nephew’s dog’s friend’s pet hamster who doesn’t know jack about endpoint security? 😏
no it’s a great point lol!
we’ve have 100% relied too much on security through obscurity. small companies could
get away with this because there is only so much surface professionals can look at.
llms change the equation now. tech debt is getting hit, along with all the vibe coded products. i don’t actually place the blame on the agents.
i mean i accomplished the goal… and like you said, they had an api endpoint with no auth.
fun times lol.
The "easy" and obvious fix is to strip software companies of the ability to avoid liability for their code behind EULAs and Ts&Cs. Microsoft making Copilot "for entertainment purposes only" is just the clowniest move, IMNSHO. 🤡 "Oh noes!1! But you used our agent for something that we didn't intend for you to do with it and therefore you can't sue us when Copilot emailed pix of Aunt Mabel in her undies to everyone in Nigeria…"
Bridge builders can't fade the liability for death and destruction based on shoddy work underneath a "contract" and neither should we in software. Will liability falling on engineers and their companies have a "chilling" effect? God, I hope so. If every semi-competent consulting company has to become liable for the code they deliver, the world will be a better place (because they'll go out of business).
totally agree. it’s been pretty interesting to see how liability hasn’t really been established yet with these incidents. bridge builders are a great metaphor here. i think we will eventually see one or two big incident ls get to the court room and that will really solidify who is responsible when these breaches and incidents happen.