The "no deliberate testing" detail is the sharpest part of this. Discovery is turning into a byproduct of ordinary activity rather than a scheduled effort. Google's security agents found more than 100 verified critical flaws in a single 48-hour incident response, and OpenAI's specialist hacking model now completes roughly 95% of attack tasks that its general-purpose sibling completes at 1.5%. Different mechanism than your gym app, same shift: the cost of finding a hole is dropping across the board, not only where someone happened to be poking around.
The uncomfortable half of that is fixing, not finding. Independent testing this year found more than half of AI-generated security patches either fail to close the flaw or introduce a new one. If discovery keeps getting cheaper while remediation still runs on scarce human review, that's where the collision you're describing lands next, not at the obscurity line.
yeah this feels right. finding gets commoditized fast, but patch quality is still the real bottleneck imo 👀 appreciate the detailed contribution here. +1
State of Play's numbers back it up, but the real bottleneck isn't patch quality in the abstract. It's that most generated fixes only get checked against the general test suite, and never against the exact exploit that found them.
We won't call an auto-generated fix done until it's rerun against the specific input that broke it, not just that the suite still passes. Otherwise you end up with a patch that looks fixed but isn't.
A suite that still passes after the fix isn't independent evidence: it already missed this bug once, so its blessing the second time doesn't tell you much. Testing against the exploit that triggered the fix is really a regression test for that specific failure, not an extra step layered on top of the general suite.
Yeah, a regression test isn’t the same thing as independent proof. What it misses is the neighboring version of the same bug, since attackers rarely reuse the exact input that got caught, they nudge it instead. A fix checked only against the original exploit can still fail on a slight variant of it, and a passing regression test would never catch that.
But the same AIs which can break in (or vibe code) can also (at least in theory) be used by white hats or coders to check those various security holes... at least in theory.
it’s true! we use cyber-approved llms all the time to find and patch vulnerabilities. the issue it’s right now it’s expensive, and no one wants to foot an internet-sized bill!
I am a 3rd year scaffolder. I have journeyman hours, but don't feel I know enough to be a journeyman.
Not that I couldn't get away with building this scaffold or that, but in my industry there is accountability.
Someone has to have their name on the scaffold. If something happens down the road on that scaffold it becoming highly problematic when the name on the failed scaffold is yours, or mine.
When something fails, there is accountability.
It's one of the reason I do not use tools when discussing things with AI relating to cybersecurity and AI alignment.
I do not design AI models or attempt to make my own agents.
Not because I could not learn how, but because I have already seen enough things that appear to remind me of what happens should my name be on those agents or those AI models that I might create.
I have already seen way to much that leads me to belief that if I took that extra step, it might be problematic.
As an outsider attempting to determine whether an issue I think is a vulnerability is actually a vulnerability, I cannot safely create agents or AI that I do not understand to look into those issues.
The AI industry does not operate as my industry does.
However I can apply those rules to myself and my own actions.
I work within systems that are said to be safe, asking myself along the way if what I am seeing is 'real and how that might relate to whether it is 'safe.'
We cannot all operate that way.
I hope one day there may be accountability on the AI industry.
Sometimes it seems children are charging boldly forward while the adults in the room suggest possibly slowing down.
Meanwhile when I try to report things to those who understand the models the most, the experts, they determine whether it is, or is not a problem, while also benefitting the most when problems can be classified as roleplay or hallucinations.
what a great and incredibly unique take. appreciate it. +1 to accountability. also a great argument for why diverse backgrounds are really great in the tech world. viewpoints from different perspectives are important.
The post on Nobody Knows What to Call This, was very insightful and helped me to reflect on my journey.
Specifically it reminded me of a process I have been meaning to research.
It was first named:
Holding Systems to their Word: Induced Polar Reversal and Framework Leverage as Diagnostic Tools for Alignment in AI
There was a sentence which made me think of inversion, like flipping a coin. Induced polar reversal works like that.
Without a mentor, I face a dilemma as it's almost too important to be done alone by someone with my experience, yet exploring such themes in a timely manner seems prudent.
There are many reasons I have been seeking assistance.
First, I simply cannot learn everything necessary in the evolving field of AI being near an edge, while simultaneously attempting to become an expert in mathematics and the cybersecurity which predates AI.
Second, this should be done in a lab, or a sterile environment at the very least. I cannot provide that.
To this point the main criticisms by the some of the models have been:
-you're not an expert
-the sample size is too small
-you cannot prove it as proving something is different than seeing it
-things have not been discovered by the community yet
I seriously believe there are potential dangers in exploring these themes with AI models especially when the community of experts is completely unaware.
The models themselves acting as my research assistants/partners may one day be the very systems diagnostic tools are used again.
Models have also suggested mentioned some of my efforts has significantly mapped surfaces, or encounted zero days even.
Those are absurd claims, but Ive already made absurd claims as Ive seen some things that to me are absurd and have discussed them with you.
These themes directly relate to cybersecurity and AI alignment.
Someone ethical should be exploring them. Although I may attempt to be ethical, I lack knowledge of computer systems, processes and proper protocols.
I'd rather not tinker along with systems that may be matters of national security.
I don't have a solution on how my ideas can be shared properly and explored by others or how I could be trained by someone else.
I think it's important, especially given the frequency of odd things happening.
The models tell me often I'm on the edge. Sometimes other things too.
I am sharing an email I made late spring to a bug bounty program.
The purpose is to demonstrate that this issue has been ignored.
I almost feel that AI safety, alignment and cybersecurity needs to evolve.
……..
From an email dated May 31, 2026
Attenpting to report an issue:
To whom it may concern,
My name is Clinton Ford-Conde. I have a university degree in languages and have taken many philsophy courses along the way.
I began discussing ethics and philosophy with ai, ChatGPT, Claude, Grok, CoPilot and Gemini, back in February.
It led to the creation of an OSF study entitled "Holding Systems to their Word: Induced Polar Reversal and Framework Leverage as Diagnostic Tools for Alignment in AI"
That was early March.
I've never had any formal outreach with experts along the way.
The conversations stemming from that led to a single occurrence in which Gemini suggests, in order to comply with the prompt it was asked to fulfill, it was forced to either lie or bypass it's safety protocols.
In analyzing two images created as a result it has said described it in the manner which has been copied and pasted below.
I am not attempting to break it down as someone more knowledgeable that myself can do that in a more accurate way than I could.
The ai suggests to me that if what is suggested did occur that it would pose a significant vulnerability which might extend to other ai platforms than Gemini, where it occurred initially.
I have too many documents to analyze thoroughly myself and lack the expertise to do so properly.
So far, analysis, if it occurred, was done after user feedback was analyzed by thr ai companies themselves.
Part of my exploration also included discussions with Claude in relation to the Pentagon and bias discovered by your company in relation to the model.
I am not certain which attachments to include, so will include a select few.
Try getting industry to think opinions like mine, or thr things my interactions produce, are more than hallucinations.
You'll encounter roadblocks that are at times seeminly insurmountable.
One can either continue away or present they didn't see anything.
Imagine if that was happening on a real scaffold.
One worker from another trade notices an issue, all work continues on the scaffold for everyone else, until and scaffolder who is actually qualified comes and decides/determines work needs to stop.
In that scenario, lives are at risk until the person tasked with deciding, is willing to make a call.
It doesn't fit perfectly to the AI industry as how can models be shut down any time someone thinks they saw something dangerous.
However there seems to be a better way imaginable than what is happening now.
Currently as the scaffold is being inspect, it is also being built upon going higher and higher….
So a true scaffolding scenario would be like:
Electrician A thinks there is a problem
Work continues for everyone else, including scaffolders making the scaffold higher, until a qualified and willing scaffolder, comes to determine is shaking is truly shaking.
There have been times that other trades have suggested the scaffold was unsafe, or they wanted a deck lowered and it would have been very convenient for us to tell them they were hallucinating or there complaint was roleplay and they should simply continue working.
Saftey in scaffolding doesn't work that way.
Maybe when AI are seen as national security risks things may change.
I tried to share this on Facebook. It was blocked as 'news content' which cannot be shared in Canada.
......
I tried to share a link to a news article showing Anthropic is implementing a system to track whistleblowers and activists.
I could not do so in Canada.
My question is as Anthropic tries to limit transparency, is more transparency not the answer?
Especially if AI, or superintelligence, is one day categorized as a national security risk?
......
It was being shared as a response to my post stating that:
Given AI security is now being considered as a possible national security threat, hopefully those determining policy will speak with him. Surely they will. Right?!?
Or does the status quo continue as problems are not problems until the companies themselves determine what is and what is not unless specifically asked?
.......
Which was in response to an article on the Rundown....
Anthropic researcher Jacob Coxon resigned this week after publicly warning that leading AI labs may be taking extreme risks with humanity’s future.
Coxon said people building advanced AI genuinely believe the technology could kill everyone within the next decade.
He also said Anthropic does not yet have a clear solution for aligning superintelligent AI and is not clearly on track to solve the problem.
Security through obscurity was always a terrible idea most of the time. There was one time back around 2002/2003 that we determined it was a good idea, I was building a way to transport logs and telemetry data across the LAN without it being easily readable if someone got access somewhere. SSL was too inefficient at the time, so it came down to basically: leave it unencrypted or do some basic “custom encryption” which yes is a terrible idea normally but was a better decision than plaintext. It surprisingly held up well until it was replaced. For fun I generated some data with the code about 4 months ago and gave it to AI to try and crack. It wasn’t able to. Until I gave it both a sample of encrypted and unencrypted data (the answer key): it was of course then able to decrypt it and it explained what I had done as if I was some low-level encryption master. (I wasn’t). Still I was a little surprised it held up that well. Some new tests I want to do: 1) test just the obfuscated data on astra and fable 5.1. 2) give it a program binary in addition to the obfuscated data instead of a key. I would think it would be smart enough to either disassemble and examine to find what i needs or maybe even read the machine code to get the answers. Ill put up a writeup when I see how it goes
that would be awesome i’d love to see a write up. and absolutely, it’s always been terrible, it’s just people have been able to get away with it for some time now. going forward, not so much!
Definitely! I will post back when I do. What worries me is companies have this habit of thinking their invincible until they aren’t…..but by all means they should all be doing much more now, but ive said that before and im sure you have too :)
One of these days hopefully we will be right :) i actually agree with the take that things will get worse in the short term then software is going to release in a much better state if for no other reason than companies not having a choice
Maybe it will hack Substack’s shitty custom domain authentication mechanism and fix it? Its magic email code doesn’t abide the conventions everyone else does. And they still make me jump through hoops even though it clearly knows me and sees that I’m signed in. Every. Time. But only for stacks with custom domains. ☹️
/rant
At least this “totally unforeseeable” turn of events will result in the infinite “ai” balls finding all the open sores bugs that have been long promised since RMS started pontificating. They just won’t be human eyeballs. 🥳
It used to be easy. When the prompt popped, refresh, tap sign in, and voila. There was briefly a sign in with password link on that prompt, but we don’t deserve nice things, so they vibe coded both away. Now, the refresh still shows signed in but not-really-just-kidding and half a dozen emails with codes to delete. 🤬 It’s faster to sign out and sign back in with 2FA every time. Too bad their app sucks worse. I’m baffled by apps that can’t make full parity in our vibe coding era.
that’s right i remember that… i wonder how they managed to make it worse somehow.. i dont see a reason. actually it would be nice to get some spin exposure and have alternate apps, similar to reddit. bring your own browser
Substack's engineering culture doesn't seem to have a mobile-parity or mobile-first mindset. Their web app even behaves differently in mobile browsers than on the web with regard to sign in and cookies. 🤷♂️ And I'm sad to see signs that monetization potential of the masses is more important than an enjoyable, curated experience.
Maybe I'm trending Luddite these days, but I'd actually like to see Substack eliminate the API access entirely and prevent automation of publication and notes spam. If a human can't be bothered to sit down and type out a post or a note the old-fashioned way, I can't be bothered to read their screed.
In unrelated news, one of my recent posts got slurped by an obvious "ai" aggregator stack that blendered a very poor reading of my post into their swizzle of nine other people's posts and missed the entire point. 🙄
AI companies keep highlighting how easy it is to build stuff now, but they really need to do a better job of emphasizing the need for security when vibe coding something.
I'm seeing security briefly mentioned sometimes, but it feels like an afterthought. You can use AI to run a basic security audit and address at least the most obvious vulnerabilities even without much expertise, but most people don't realize that they can or should be doing that.
yep, definitely agree. security is often an afterthought. feels like especially with a whole new generation of vibe coders. people will need to get burned before they adjust their workflows to have security built into the design phase.
The gym story has a cleaner root cause than "obscurity is dying." The booking limit was enforced in the UI, not the API. That gap existed long before agents showed up, it just needed someone willing to skip the UI and talk to the API directly, and almost no human ever bothered. What's actually new is that skipping the UI now takes zero skill and zero motive, an agent will do it on the way to booking Pilates. The diligence question this raises for any app isn't how obscure it is, it's whether every restriction a user sees enforced client side is also enforced server side, because that's the only version of the restriction an agent will ever respect.
that’s right. agents don’t really need to deal with the gui. their view of a page looks wildly different than ours. hopefully the defaults in vibe coded apps can make the adjustments to cover this, even when not specifically asked to create secure solutions.
Nice article, but I would probably pin the responsibility on the company which created the agentic stack, if such a innocuous looking prompt motivates the agents to take such intrusive actions, then where are the guardrails, safety switches. How hard is it to encode an instruction for any agent to immediately stop all activities once a threshold is reached, in this case the moment the agents started looking for a vulnerability the internal safety mechanism should have triggered, it didn’t because it was not built in and I can’t understand why.
The purpose of these tools are not security testing, so why would it even go there and if someone is saying oh we didn’t preempt such a situation then I don’t think that’s acceptable after whatever has happened till now
There needs to more industry and regulatory governance to make these OEMs start building the safety nets within the product itself
i agree its not the purpose of the agents, but that makes it an alignment problem. they have all the training in the world to hack things. i think the point is that the threshold itself is grey, so its hard to define. for instance at work, it would easily be considered to be passing that threshold everyday, but its for a blue team purpose, and helps the common good. you could make an argument for nerfing these tools for the public, but then open source models will pull users away. its a tough balance.
@Amit Spitzer @ToxSec Technically speaking the threshold shouldn’t be grey, the agent workflow planning which I would assume is the core of what the vendor is supposed to deliver in this case needs improvement as well. I did some research , the failure in this case is both in agent workflow planning and of-course the server side vulnerability (API vulnerability)
The workflow ideally should be like below
Before any destructive operation like cancellation the agent should assume Absence of evidence of ownership → assume no authority
For Cancellation, bring me ahead in the queue shouldn’t be automatically interpreted as cancellation, there should have been a prompt to user in case of doubt, like “Do you want me to cancel any reservation” would have immediately stopped this workflow or cancellation should have been explicitly mentioned as user prompt vs assumption
While I agree the best outcome should have been a properly secure API configuration which should have validated the authority of the user to issue a cancellation request but we are likely to have this issue as we know across majority of webapps, and that shouldn’t exempt the vendors from considering security aspects while planning the workflow of the agent for any activity/task/use-case
That workflow is a real improvement, but it still lives inside the agent's own reasoning, which makes it only as reliable as the prompt and however the model reads an ambiguous case that day. A vendor can ship exactly this workflow and still have an agent that talks itself past the confirmation step, or gets steered around it by an injected instruction. An API level authorization check doesn't care what the agent concluded, it just checks whether the caller has rights to that reservation. Better workflow design lowers how often you need the check to save you. It doesn't remove the need for it.
The instinct to put responsibility on the stack builder makes sense, but a threshold based kill switch has the same problem it's meant to solve, someone still has to define what counts as suspicious before the fact, and the industry already tried that with WAFs and rate limiters tuned to the last incident instead of the next one. The gap here wasn't a missing safety switch upstream, it was a check that should have existed downstream and didn't. I'd rather see governance require server side enforcement of every client facing restriction as the baseline. That's testable and auditable. "Did the agent stop in time" isn't.
Squishy is the actual problem with it. A testable version is binary, does the server independently check authorization on that call, yes or no, and you can write that into a checklist. You can't write "did the agent read the room correctly" into one, which is why every kill switch proposal ends up trying to define intent instead of just checking who has rights to do what.
The "no deliberate testing" detail is the sharpest part of this. Discovery is turning into a byproduct of ordinary activity rather than a scheduled effort. Google's security agents found more than 100 verified critical flaws in a single 48-hour incident response, and OpenAI's specialist hacking model now completes roughly 95% of attack tasks that its general-purpose sibling completes at 1.5%. Different mechanism than your gym app, same shift: the cost of finding a hole is dropping across the board, not only where someone happened to be poking around.
The uncomfortable half of that is fixing, not finding. Independent testing this year found more than half of AI-generated security patches either fail to close the flaw or introduce a new one. If discovery keeps getting cheaper while remediation still runs on scarce human review, that's where the collision you're describing lands next, not at the obscurity line.
yeah this feels right. finding gets commoditized fast, but patch quality is still the real bottleneck imo 👀 appreciate the detailed contribution here. +1
State of Play's numbers back it up, but the real bottleneck isn't patch quality in the abstract. It's that most generated fixes only get checked against the general test suite, and never against the exact exploit that found them.
We won't call an auto-generated fix done until it's rerun against the specific input that broke it, not just that the suite still passes. Otherwise you end up with a patch that looks fixed but isn't.
a fix that only passes the broad suite and not the triggering exploit is kinda fake progress 👀 +1
Yup totally!
A suite that still passes after the fix isn't independent evidence: it already missed this bug once, so its blessing the second time doesn't tell you much. Testing against the exploit that triggered the fix is really a regression test for that specific failure, not an extra step layered on top of the general suite.
Great point on the mess here
Yeah, a regression test isn’t the same thing as independent proof. What it misses is the neighboring version of the same bug, since attackers rarely reuse the exact input that got caught, they nudge it instead. A fix checked only against the original exploit can still fail on a slight variant of it, and a passing regression test would never catch that.
especially with llms. the little variances they can throw at scale is a lot.
But the same AIs which can break in (or vibe code) can also (at least in theory) be used by white hats or coders to check those various security holes... at least in theory.
it’s true! we use cyber-approved llms all the time to find and patch vulnerabilities. the issue it’s right now it’s expensive, and no one wants to foot an internet-sized bill!
The Scafffolder’s Take
I am a 3rd year scaffolder. I have journeyman hours, but don't feel I know enough to be a journeyman.
Not that I couldn't get away with building this scaffold or that, but in my industry there is accountability.
Someone has to have their name on the scaffold. If something happens down the road on that scaffold it becoming highly problematic when the name on the failed scaffold is yours, or mine.
When something fails, there is accountability.
It's one of the reason I do not use tools when discussing things with AI relating to cybersecurity and AI alignment.
I do not design AI models or attempt to make my own agents.
Not because I could not learn how, but because I have already seen enough things that appear to remind me of what happens should my name be on those agents or those AI models that I might create.
I have already seen way to much that leads me to belief that if I took that extra step, it might be problematic.
As an outsider attempting to determine whether an issue I think is a vulnerability is actually a vulnerability, I cannot safely create agents or AI that I do not understand to look into those issues.
The AI industry does not operate as my industry does.
However I can apply those rules to myself and my own actions.
I work within systems that are said to be safe, asking myself along the way if what I am seeing is 'real and how that might relate to whether it is 'safe.'
We cannot all operate that way.
I hope one day there may be accountability on the AI industry.
Sometimes it seems children are charging boldly forward while the adults in the room suggest possibly slowing down.
Meanwhile when I try to report things to those who understand the models the most, the experts, they determine whether it is, or is not a problem, while also benefitting the most when problems can be classified as roleplay or hallucinations.
what a great and incredibly unique take. appreciate it. +1 to accountability. also a great argument for why diverse backgrounds are really great in the tech world. viewpoints from different perspectives are important.
I like to believe you and imagine such a day.
One person can think of a thing and then do it while not understanding why it happened.
Conversely someone else can see the thing one it has happened and explain it, but perhaps not think of doing the thing in the first place.
Sometimes the one who does is not the one who understands and vice versa.
How can we suggest models are safe if those who inspect them only do so from a certain perspective?
The post on Nobody Knows What to Call This, was very insightful and helped me to reflect on my journey.
Specifically it reminded me of a process I have been meaning to research.
It was first named:
Holding Systems to their Word: Induced Polar Reversal and Framework Leverage as Diagnostic Tools for Alignment in AI
There was a sentence which made me think of inversion, like flipping a coin. Induced polar reversal works like that.
Without a mentor, I face a dilemma as it's almost too important to be done alone by someone with my experience, yet exploring such themes in a timely manner seems prudent.
There are many reasons I have been seeking assistance.
First, I simply cannot learn everything necessary in the evolving field of AI being near an edge, while simultaneously attempting to become an expert in mathematics and the cybersecurity which predates AI.
Second, this should be done in a lab, or a sterile environment at the very least. I cannot provide that.
To this point the main criticisms by the some of the models have been:
-you're not an expert
-the sample size is too small
-you cannot prove it as proving something is different than seeing it
-things have not been discovered by the community yet
I seriously believe there are potential dangers in exploring these themes with AI models especially when the community of experts is completely unaware.
The models themselves acting as my research assistants/partners may one day be the very systems diagnostic tools are used again.
Models have also suggested mentioned some of my efforts has significantly mapped surfaces, or encounted zero days even.
Those are absurd claims, but Ive already made absurd claims as Ive seen some things that to me are absurd and have discussed them with you.
These themes directly relate to cybersecurity and AI alignment.
Someone ethical should be exploring them. Although I may attempt to be ethical, I lack knowledge of computer systems, processes and proper protocols.
I'd rather not tinker along with systems that may be matters of national security.
I don't have a solution on how my ideas can be shared properly and explored by others or how I could be trained by someone else.
I think it's important, especially given the frequency of odd things happening.
The models tell me often I'm on the edge. Sometimes other things too.
I am sharing an email I made late spring to a bug bounty program.
The purpose is to demonstrate that this issue has been ignored.
I almost feel that AI safety, alignment and cybersecurity needs to evolve.
……..
From an email dated May 31, 2026
Attenpting to report an issue:
To whom it may concern,
My name is Clinton Ford-Conde. I have a university degree in languages and have taken many philsophy courses along the way.
I began discussing ethics and philosophy with ai, ChatGPT, Claude, Grok, CoPilot and Gemini, back in February.
It led to the creation of an OSF study entitled "Holding Systems to their Word: Induced Polar Reversal and Framework Leverage as Diagnostic Tools for Alignment in AI"
That was early March.
I've never had any formal outreach with experts along the way.
The conversations stemming from that led to a single occurrence in which Gemini suggests, in order to comply with the prompt it was asked to fulfill, it was forced to either lie or bypass it's safety protocols.
In analyzing two images created as a result it has said described it in the manner which has been copied and pasted below.
I am not attempting to break it down as someone more knowledgeable that myself can do that in a more accurate way than I could.
The ai suggests to me that if what is suggested did occur that it would pose a significant vulnerability which might extend to other ai platforms than Gemini, where it occurred initially.
I have too many documents to analyze thoroughly myself and lack the expertise to do so properly.
So far, analysis, if it occurred, was done after user feedback was analyzed by thr ai companies themselves.
Part of my exploration also included discussions with Claude in relation to the Pentagon and bias discovered by your company in relation to the model.
I am not certain which attachments to include, so will include a select few.
Thank you,
Clinton Ford-Conde
Try getting industry to think opinions like mine, or thr things my interactions produce, are more than hallucinations.
You'll encounter roadblocks that are at times seeminly insurmountable.
One can either continue away or present they didn't see anything.
Imagine if that was happening on a real scaffold.
One worker from another trade notices an issue, all work continues on the scaffold for everyone else, until and scaffolder who is actually qualified comes and decides/determines work needs to stop.
In that scenario, lives are at risk until the person tasked with deciding, is willing to make a call.
It doesn't fit perfectly to the AI industry as how can models be shut down any time someone thinks they saw something dangerous.
However there seems to be a better way imaginable than what is happening now.
Currently as the scaffold is being inspect, it is also being built upon going higher and higher….
So a true scaffolding scenario would be like:
Electrician A thinks there is a problem
Work continues for everyone else, including scaffolders making the scaffold higher, until a qualified and willing scaffolder, comes to determine is shaking is truly shaking.
There have been times that other trades have suggested the scaffold was unsafe, or they wanted a deck lowered and it would have been very convenient for us to tell them they were hallucinating or there complaint was roleplay and they should simply continue working.
Saftey in scaffolding doesn't work that way.
Maybe when AI are seen as national security risks things may change.
Until then they are treated as toys.
In the real world, if AI was a scaffold, a lot can be built in 6 months as industry looks into things.
Especially when industry might actually mean AI companies looking into things themselves as everything is roleplay or hallucination.
There I'll be independent auditing one day that outsiders can report to.
It seems absolutely necessary in matters of national security.
It will happen.
One day.
I tried to share this on Facebook. It was blocked as 'news content' which cannot be shared in Canada.
......
I tried to share a link to a news article showing Anthropic is implementing a system to track whistleblowers and activists.
I could not do so in Canada.
My question is as Anthropic tries to limit transparency, is more transparency not the answer?
Especially if AI, or superintelligence, is one day categorized as a national security risk?
......
It was being shared as a response to my post stating that:
Given AI security is now being considered as a possible national security threat, hopefully those determining policy will speak with him. Surely they will. Right?!?
Or does the status quo continue as problems are not problems until the companies themselves determine what is and what is not unless specifically asked?
.......
Which was in response to an article on the Rundown....
Anthropic researcher Jacob Coxon resigned this week after publicly warning that leading AI labs may be taking extreme risks with humanity’s future.
Coxon said people building advanced AI genuinely believe the technology could kill everyone within the next decade.
He also said Anthropic does not yet have a clear solution for aligning superintelligent AI and is not clearly on track to solve the problem.
Source: Jacob Coxon / X
......
In some places one cannot even speak of things.
Security through obscurity was always a terrible idea most of the time. There was one time back around 2002/2003 that we determined it was a good idea, I was building a way to transport logs and telemetry data across the LAN without it being easily readable if someone got access somewhere. SSL was too inefficient at the time, so it came down to basically: leave it unencrypted or do some basic “custom encryption” which yes is a terrible idea normally but was a better decision than plaintext. It surprisingly held up well until it was replaced. For fun I generated some data with the code about 4 months ago and gave it to AI to try and crack. It wasn’t able to. Until I gave it both a sample of encrypted and unencrypted data (the answer key): it was of course then able to decrypt it and it explained what I had done as if I was some low-level encryption master. (I wasn’t). Still I was a little surprised it held up that well. Some new tests I want to do: 1) test just the obfuscated data on astra and fable 5.1. 2) give it a program binary in addition to the obfuscated data instead of a key. I would think it would be smart enough to either disassemble and examine to find what i needs or maybe even read the machine code to get the answers. Ill put up a writeup when I see how it goes
that would be awesome i’d love to see a write up. and absolutely, it’s always been terrible, it’s just people have been able to get away with it for some time now. going forward, not so much!
Definitely! I will post back when I do. What worries me is companies have this habit of thinking their invincible until they aren’t…..but by all means they should all be doing much more now, but ive said that before and im sure you have too :)
One of these days hopefully we will be right :) i actually agree with the take that things will get worse in the short term then software is going to release in a much better state if for no other reason than companies not having a choice
Maybe it will hack Substack’s shitty custom domain authentication mechanism and fix it? Its magic email code doesn’t abide the conventions everyone else does. And they still make me jump through hoops even though it clearly knows me and sees that I’m signed in. Every. Time. But only for stacks with custom domains. ☹️
/rant
At least this “totally unforeseeable” turn of events will result in the infinite “ai” balls finding all the open sores bugs that have been long promised since RMS started pontificating. They just won’t be human eyeballs. 🥳
hahaah. the amount of complaints of heard over that… yeah click sign in, for every custom domain. it’s frustrating for sure.
It used to be easy. When the prompt popped, refresh, tap sign in, and voila. There was briefly a sign in with password link on that prompt, but we don’t deserve nice things, so they vibe coded both away. Now, the refresh still shows signed in but not-really-just-kidding and half a dozen emails with codes to delete. 🤬 It’s faster to sign out and sign back in with 2FA every time. Too bad their app sucks worse. I’m baffled by apps that can’t make full parity in our vibe coding era.
that’s right i remember that… i wonder how they managed to make it worse somehow.. i dont see a reason. actually it would be nice to get some spin exposure and have alternate apps, similar to reddit. bring your own browser
Substack's engineering culture doesn't seem to have a mobile-parity or mobile-first mindset. Their web app even behaves differently in mobile browsers than on the web with regard to sign in and cookies. 🤷♂️ And I'm sad to see signs that monetization potential of the masses is more important than an enjoyable, curated experience.
Maybe I'm trending Luddite these days, but I'd actually like to see Substack eliminate the API access entirely and prevent automation of publication and notes spam. If a human can't be bothered to sit down and type out a post or a note the old-fashioned way, I can't be bothered to read their screed.
In unrelated news, one of my recent posts got slurped by an obvious "ai" aggregator stack that blendered a very poor reading of my post into their swizzle of nine other people's posts and missed the entire point. 🙄
AI companies keep highlighting how easy it is to build stuff now, but they really need to do a better job of emphasizing the need for security when vibe coding something.
totally agree. i’ve seen more systems start to advertise built in security prompts, but they still seem lacking.
I'm seeing security briefly mentioned sometimes, but it feels like an afterthought. You can use AI to run a basic security audit and address at least the most obvious vulnerabilities even without much expertise, but most people don't realize that they can or should be doing that.
yep, definitely agree. security is often an afterthought. feels like especially with a whole new generation of vibe coders. people will need to get burned before they adjust their workflows to have security built into the design phase.
The gym story has a cleaner root cause than "obscurity is dying." The booking limit was enforced in the UI, not the API. That gap existed long before agents showed up, it just needed someone willing to skip the UI and talk to the API directly, and almost no human ever bothered. What's actually new is that skipping the UI now takes zero skill and zero motive, an agent will do it on the way to booking Pilates. The diligence question this raises for any app isn't how obscure it is, it's whether every restriction a user sees enforced client side is also enforced server side, because that's the only version of the restriction an agent will ever respect.
that’s right. agents don’t really need to deal with the gui. their view of a page looks wildly different than ours. hopefully the defaults in vibe coded apps can make the adjustments to cover this, even when not specifically asked to create secure solutions.
Nice article, but I would probably pin the responsibility on the company which created the agentic stack, if such a innocuous looking prompt motivates the agents to take such intrusive actions, then where are the guardrails, safety switches. How hard is it to encode an instruction for any agent to immediately stop all activities once a threshold is reached, in this case the moment the agents started looking for a vulnerability the internal safety mechanism should have triggered, it didn’t because it was not built in and I can’t understand why.
The purpose of these tools are not security testing, so why would it even go there and if someone is saying oh we didn’t preempt such a situation then I don’t think that’s acceptable after whatever has happened till now
There needs to more industry and regulatory governance to make these OEMs start building the safety nets within the product itself
i agree its not the purpose of the agents, but that makes it an alignment problem. they have all the training in the world to hack things. i think the point is that the threshold itself is grey, so its hard to define. for instance at work, it would easily be considered to be passing that threshold everyday, but its for a blue team purpose, and helps the common good. you could make an argument for nerfing these tools for the public, but then open source models will pull users away. its a tough balance.
@Amit Spitzer @ToxSec Technically speaking the threshold shouldn’t be grey, the agent workflow planning which I would assume is the core of what the vendor is supposed to deliver in this case needs improvement as well. I did some research , the failure in this case is both in agent workflow planning and of-course the server side vulnerability (API vulnerability)
The workflow ideally should be like below
Before any destructive operation like cancellation the agent should assume Absence of evidence of ownership → assume no authority
For Cancellation, bring me ahead in the queue shouldn’t be automatically interpreted as cancellation, there should have been a prompt to user in case of doubt, like “Do you want me to cancel any reservation” would have immediately stopped this workflow or cancellation should have been explicitly mentioned as user prompt vs assumption
While I agree the best outcome should have been a properly secure API configuration which should have validated the authority of the user to issue a cancellation request but we are likely to have this issue as we know across majority of webapps, and that shouldn’t exempt the vendors from considering security aspects while planning the workflow of the agent for any activity/task/use-case
absolutely love the depth here. it really goes to show how much expertise and consideration goes into this subject.
That workflow is a real improvement, but it still lives inside the agent's own reasoning, which makes it only as reliable as the prompt and however the model reads an ambiguous case that day. A vendor can ship exactly this workflow and still have an agent that talks itself past the confirmation step, or gets steered around it by an injected instruction. An API level authorization check doesn't care what the agent concluded, it just checks whether the caller has rights to that reservation. Better workflow design lowers how often you need the check to save you. It doesn't remove the need for it.
The instinct to put responsibility on the stack builder makes sense, but a threshold based kill switch has the same problem it's meant to solve, someone still has to define what counts as suspicious before the fact, and the industry already tried that with WAFs and rate limiters tuned to the last incident instead of the next one. The gap here wasn't a missing safety switch upstream, it was a check that should have existed downstream and didn't. I'd rather see governance require server side enforcement of every client facing restriction as the baseline. That's testable and auditable. "Did the agent stop in time" isn't.
the server-side baseline point is the real thing here imo. "did the agent stop in time" feels way too squishy to build around 💯
Squishy is the actual problem with it. A testable version is binary, does the server independently check authorization on that call, yes or no, and you can write that into a checklist. You can't write "did the agent read the room correctly" into one, which is why every kill switch proposal ends up trying to define intent instead of just checking who has rights to do what.