Share from 0:00
0:00
/

AI Agents Can Spend Your Money. What Could Go Wrong?

AI agents are getting real power. They can spend your money, manage your accounts, find security vulnerabilities, and even write the patches to fix them. But what happens when something goes wrong?

An AI agent browsed emails for me and found a subscription discount. After my permission, it logged in, accessed a verification email, and changed my subscription level, saving me about $8 a month (from ~$20 to $12).

Agents are useful. But useful things we should be wary of?


On this week’s podcast episode about agents making payments we discussed how I would want an agent to have some sort of prepaid wallet, like give it $500 and see what it does with it (and once it’s gone it can’t do anything else!).

What’s interesting to me is that as agents become more useful and do more tasks for us successfully, we’re probably going to give them more permissions to act on our behalf (make changes, make purchases, etc.) without asking us each and every time. This is probably a good thing, because otherwise what’s the point? But it also means that when something goes wrong it’ll have more latitude to wreak havoc.

Granted, in the above scenario there were a couple of different permissions being granted… read my email, log into this service, change my subscription. It’s just that the service made it seem like one seamless process. But these permissions have different security implications, and should be thought about separately.

This is something OpenAI thought about when they recently announced ChatGPT Wallet. In the initial launch, payments made through ChatGPT Wallet will require user approval (there’s a security checkpoint). They’ll use payment tokens instead of card data so ChatGPT (and the merchant) won’t have access to your card data.

So, ChatGPT can help you find things you want to buy, but there’s another process involved in actually buying them. In the future though OpenAI envisions people setting limits (time based, purchase category, amount, etc.) and pre-approving purchases.

The interesting security question to me is then, how are these limits implemented and enforced?

Share ToxSec - AI and Cybersecurity


Successful ≠ Secure

Imagine you’re trying out an AI agent, and you give it a bunch of tasks to do to see how well it performs. It does well! So you decide to give it access to your GitHub account to see if it can help troubleshoot an issue.

Sounds reasonable. You vetted the AI agent by having it do a bunch of tasks and it did well. So you give it more access, right?

Well, what if during vetting you only learned that it was good at doing the tasks you gave it? What if there was some hidden behavior that you didn’t uncover during vetting? What if that hidden behavior got triggered after you gave the AI agent access to your GitHub account?

This week Microsoft published research (Lazarus) showing how they were able to plant behaviors in models by tweaking a small number of neurons. In one example they tweaked 7 neurons (out of billions!) in the last feed-forward layer of a model (Nemotron-Terminal-8B), and the implanted behavior was successful in 98.8% of cases (when the trigger was included).

Granted, the planted behavior in this case was benign (just predicting a specific string of tokens - a marker - when presented with a trigger string of tokens). But it shows how a model can appear to be working normally but exhibit unexpected behavior when presented with certain inputs. Granted, this particular attack required the attacker to have access to tweak the model’s parameters (so think compromised model in the supply chain). But still worth thinking about.

Microsoft also discussed techniques to identify when models’ parameters have been tampered with.

The main point here is that just because a model does well in some contexts (passes your vetting process), doesn’t mean it won’t misbehave in other contexts. So it’s worth thinking about where models come from, validating their integrity, and limiting their access to your systems.

7 neurons can have big implications

So let’s say you did validate a model’s integrity and limited its access to your systems. What if it still spat out some behavior you didn’t expect?

Well in that case it’s helpful to think about the different safeguards you have in place - permissions, sandboxing, network access, etc. These factors will determine if the behavior leads to nothing (safeguards worked!), a nuisance (maybe broke some things in a non-prod system), or a bigger issue (impacted prod).

The number of tweaked neurons made for an interesting research paper title. But the impact of that behavior in production is what’ll determine if it makes for an interesting war story.

Leave a comment


Let’s give it just a little more access…

AI agents are getting pretty good at identifying software vulnerabilities. But after that there’s still a bunch of work to do - validate the vuln, figure out where the issue is, develop a fix, test the fix, deploy the fix, etc. So even when AI agents identify juicy vulns it could take days or weeks to get someone to circle back and address the issue.

So what if AI agents could help with that work too? OpenAI recently talked about their Daybreak Defense Factory - AI agents identify vulns (either from traditional security solutions or Daybreak Red Team) and propose patches, which engineers can review and validate before merging.

Imagine if every time an AI agent identified a vuln it also provided a proposed patch and evidence showing why the patch should address the vuln. That’s pretty neat! Engineers could spend their time reviewing patches instead of wading through codebases to figure out where issues are and how to address them.

So let’s say you’re an engineer tasked with reviewing patches proposed by AI agents. Seems like it’d be helpful for the AI agent to have access to your codebase to understand where issues are and develop patches. And maybe it’s helpful to give the agent access to a dev environment so it can test out its patch and gather evidence to include along with the patch it submits for your review.

Seems pretty reasonable.

What about giving the agent access to deploy the patch once you approve it? That might save some time. What about trusting the agent to make the call on whether or not its patch should be deployed based on the results from testing it in the dev environment? That might save some time too…

AI agents can be helpful defenders. But it’s worth thinking through what access they need to be helpful defenders, what guardrails make sense to have in place (How will their work be validated? How quickly could their actions be rolled back if needed?), and making sure we don’t accidentally give them more access than we intended to. Otherwise we risk AI agents outpacing our ability to keep up with what they’re doing and we’ll start to get pressure to lower guardrails to keep up with all the good things they’re doing…

Thanks for reading ToxSec! This post is public so feel free to share it.

Share


Set limits before you need them

Back when I was thinking about guardrails for agentic payments I suggested giving agents prepaid cards with a fixed amount of money that couldn’t be automatically recharged as a way to guardrail agents’ ability to spend. The idea being that even if something went sideways the agent could only spend so much.

This week OpenAI introduced ChatGPT Wallet. In addition to offering guardrails like setting limits on how much agents can spend and requiring your OK before agents can use new merchants, OpenAI mentions prepaid balances as a way customers can limit ChatGPT’s spending.

I like prepaid cards as a guardrail because they create a hard limit on an agent’s ability to spend money, which is not something the agent can try to talk you (or itself) out of. Telling an agent in its system prompt “Don’t spend more than $500!” technically sets a limit on the agent’s ability to spend money, but it’s a limit enforced by… the agent.

Of course prepaid cards don’t address every risk - agents with access to your email could still exfiltrate confidential info, agents with access to your computer could still delete important files, etc. But it does highlight the importance of thinking about what access agents need to accomplish tasks you want them to do and implementing guardrails appropriate for each type of access:

  • Spending money - spending limits, pre-approved merchants, requiring your OK to add new merchants, etc.

  • Making code changes - can submit PRs but changes go through normal release management process before being deployed

  • Managing accounts - give agents temporary, revocable access to perform specific tasks

Etc, etc, etc

The specific guardrails that make sense will vary depending on the task(s) you want agents to perform, the systems they need access to perform those tasks, how you want work performed by agents validated, how quickly you want to be able to roll back actions taken by agents, etc. But thinking through those questions up front can help avoid situations where agents are doing cool stuff and we realize oh crap we gave them more access than we meant to…

Look for the sleeper agents

This week Microsoft published research on methods they developed to identify “sleeper agent” behavior hidden in AI models. Lazare, veni foras shared related research on introducing sleeper agent behavior into AI models by reprogramming “dormant” neurons.

I don’t have enough of an understanding of the research to offer much insight beyond yikes. Kudos to Microsoft and Lazare, veni foras for their research though!

More AI agents doing more stuff with more access

AI agents can save me $8 on a grocery order. AI agents can help identify and patch software vulnerabilities before bad guys exploit them.

More AI agents doing more stuff with more access. Pretty neat!

Just make sure we’re intentional about the types of access agents need to be able to do neat stuff so when (not if) they do not so neat stuff we’re OK with the consequences.


Thank you Karo (Product with Attitude), Erich Winkler, Linda Aldrich and many others for tuning into our live video with Exploring ChatGPT!

Discussion about this video

User's avatar

Ready for more?