Hello everyone! First, thanks for all your patience. I’ve been moving across the country, and more articles will be on the way again soon. For today, a familiar voice is back on ToxSec.
This is Mohib’s third piece here, and this time they’re digging into AI-powered phishing, how generative AI is changing the economics of social engineering, and why the old “look for typos” advice is aging pretty badly.
» Want to be featured on ToxSec? «
Reach out over Substack and we can discuss. All ideas are welcome, if you have an article on security, let’s talk.
In our previous collaboration, we looked at AI data poisoning, a problem that doesn’t get nearly enough attention. With so many AI models now available to download and run, especially within organizations, a poisoned model can create serious security risks before anyone realizes what happened.
This time, we’re looking at AI-powered phishing.
Phishing has been around for a very long time, and it has always been closely tied to social engineering. I covered the mechanics of social engineering and how attackers manipulate people in a separate SK NEXUS piece. I’ve also worked with Joel Salinas on a previous collaboration looking at AI-powered scams. This piece goes deeper into one specific part of that problem aka how generative AI is changing phishing.
One example shows how convincing these attacks can become.
In January 2024, an employee at Arup, the engineering firm behind the Sydney Opera House, received an email from someone claiming to be the company’s UK CFO. The employee was suspicious and asked for a video call to verify the request.
The call looked convincing. The CFO and several other colleagues appeared on screen, and they looked and sounded like people the employee knew. He eventually made 15 wire transfers totaling $25.6 million to five bank accounts in Hong Kong.
Everyone on the call except the employee was an AI-generated deepfake. The attackers had used publicly available video and audio of Arup executives to create them. Arup’s CIO later described the incident as technology-enhanced social engineering.
The Arup case shows where phishing is heading. Generative AI can make fake messages and fake conversations much more convincing, while also making them easier to produce at scale.
In this piece, we’ll look at how AI is changing phishing, what automated personalization looks like at scale, why older security filters can struggle with these attacks, and what organizations and individuals can do to detect them.
Let’s get started.
How Generative AI Rewrote the Economics of Phishing
Email security training has spent two decades teaching people to spot the same tells. Awkward phrasing and generic greetings. Spelling errors and sender addresses that don’t quite match. Those tells depended on phishing being written by someone working outside their native language, under time pressure, at volume.
Generative AI removes every one of those constraints at once.
Microsoft’s 2025 Digital Defense Report found that AI-generated phishing emails achieve a 54% click-through rate, compared to 12% for manually written phishing. Microsoft describes it as the most significant change in phishing the company observed in the past year. IBM’s X-Force research found generative AI has cut the time required to draft a convincing phishing email from roughly 16 hours to about five minutes. That is close to a 200x increase in attacker output per hour worked.
Similarly, Hoxhunt’s 2026 Phishing Trends Report tracked the share of AI-assisted phishing in its detection network rising from under 5% in November 2025 to 56% in December, a 14-fold jump in a single month. That share has since settled closer to 40%.
What Automated Personalization Looks Like
Spear phishing has always required research. Learning who a target reports to. What vendors a company uses. What an executive’s writing style sounds like. That research used to be the bottleneck, but that’s not the case anymore.
A general-purpose language model can draft a fluent, contextually appropriate email in seconds. Purpose-built criminal tools go further by adapting models for specific attacks.
WormGPT, first sold on underground forums in 2023, was fine-tuned for phishing templates, malware code, and exploit writeups and stripped of the safety guardrails found in commercial models. Multiple successor versions have appeared since then. Newer variants run on commercial open-weight models and are marketed on Telegram, with subscription prices starting at around €60.
A 2026 successor called KawaiiGPT is distributed freely on GitHub. It reportedly takes about five minutes to configure and can generate a functional spear-phishing lure on request.
These tools do not need to invent information about a target. They assemble it through LinkedIn connections, recent company announcements, executive names pulled from public filings, and writing samples scraped from published emails or interviews all feed into a message that references real people and real context.
The output reads like it was written by someone who actually knows the organization. In a functional sense, it was.
Deepfake audio and video extend the same personalization into real-time interaction. Arup’s CIO later said that out of curiosity, he tried deepfaking himself using free, open-source tools after the incident. It took him about 45 minutes.
Why Signature-Based Email Filters Can’t See This Threat
Traditional email security operates on pattern recognition. That approach assumes an attacker’s output looks different from legitimate mail in some detectable way.
AI-generated phishing can bypass many traditional email defenses. Some campaigns pass DMARC checks and come from compromised legitimate accounts, making the sender appear genuine. Business email compromise can also contain no attachment or malicious link, leaving signature-based filters with little to detect.
Polymorphic campaigns make detection harder by changing subject lines, sender names, and message structures across different emails. KnowBe4’s 2025 phishing research found this type of variation across observed attacks.
The final hurdle that must be crossed is the behavioral one. In many instances, traditional email filters have judged emails based on each single message and not according to whether the request is consistent with the behavior of the user. That makes plausible requests from legitimate accounts particularly difficult to identify.
What Companies Can Do About It
No single control stops AI-generated phishing. The organizations best positioned against it are combining behavioral detection, authentication that can’t be phished, verification procedures built for deepfakes, and training that teaches judgment rather than pattern-spotting.
Shift Detection From Content to Behavior
Security vendors are increasingly focusing on communication patterns rather than message content alone. Behavioral detection systems learn how a sender typically communicates, including the types of requests they make and when and how they usually send them. The system can then flag unusual behavior even when the message contains no obvious technical warning signs.
Deploy Phishing-Resistant Authentication
Phishing-resistant multi-factor authentication, including FIDO2 security keys and passkeys, addresses a key weakness in credential-based attacks. Even if an employee enters their credentials on a convincing fake login page, those credentials cannot be reused to access the real account. This makes it much harder for adversary-in-the-middle phishing kits to capture and reuse valid authentication credentials.
Separate Verification Channels for Financial Requests
For deepfake-enabled attacks such as the one reported at Arup, the recommended controls are largely established. Channel separation in payment authorization was already recommended for business email compromise before deepfakes became a concern.
It requires a payment request received through one channel to be verified through a separate, independently initiated channel. Deepfakes add a new challenge because seeing and hearing someone on a video call can no longer, by itself, serve as independent verification.
Retrain Employees Around Procedure, Not Typos
Programs that focus on typos and generic greetings may be less effective as AI-generated phishing becomes more polished. Awareness training can instead focus on procedural discipline, such as verifying unusual requests through a second channel regardless of how convincing the original message appears. Manufactured urgency can also be treated as a reason to pause and verify a request.
These controls are already familiar to many security teams. Organizations that require out-of-band verification for wire transfers and use phishing-resistant MFA may be better positioned to handle AI-generated phishing than those that rely mainly on employees spotting poor grammar or obvious mistakes.
Don’t Underestimate Phishing
AI-powered phishing shouldn’t be taken lightly. It doesn’t always take a sophisticated attack to cause serious damage. Sometimes all it takes is a convincing email, a familiar name, or a message that creates enough urgency for someone to act without checking.
That’s why understanding social engineering matters. Phishing has always relied on manipulating people, and AI is making those manipulations easier to produce and harder to spot. I’ve covered how social engineering works and how attackers use it in a separate SK NEXUS piece.
If you want to go deeper into this space, I’d also recommend following people like Erich Winkler, Digital-Mark, Secrets of Privacy, Tate Jarrow, and Cyber Hermitca. They cover privacy and related topics from different angles, and seeing how different people approach the same problems can help you build a better understanding of what to watch for.
Thanks for reading.
See you next time.
Big thanks to Mohib for coming back for round three.
AI keeps making phishing cheaper, faster, and more convincing, but a lot of the defenses still come down to boring fundamentals: strong authentication, independent verification, and slowing down when something feels urgent.
Go give SK NEXUS a follow if you enjoyed this one!
ICYMI
In case you missed it, ToxSec coauthored a piece with Karo (Product with Attitude) on managing agentic swarms:
ToxSec was also featured in Jeff Morhous’s The AI-Augmented Engineer.
And finally, thanks to Andrei Savine and Nelson Lopes for their constant support here on Substack.
Thanks again to Mohib for writing another great guest piece! This is another one of the important issues that we will be seeing more of in the coming years!
What are your thoughts Mohib on using a password manager as one of the tools to prevent a successful credential Phishing attack? If the web URL is even slightly wrong, the password manager won't fill in the login credentials, alerting the victim to the fact that something phishy is going on.
Obviously you need multiple layers of protection to keep yourself safe, but I believe that a password manager should be one of those layers.