Researchers found a way to recover encrypted AI reasoning from Claude, GPT, and Gemini by using weaker models as decoders, exposing hidden thoughts, credentials, and a new attack surface.
Thanks for sharing this! It's really interesting how the researchers were able to get past the encryption. I get why big AI labs would do this to counter distillation, but it feels like they should’ve tested how well encrypting the chain-of-thought actually works more carefully.
it’s a little tricky if you don’t have a cryptography background. but they essentially just misconfigured their backend. imagine the post office giving you a key that opens box 111. except it opens all the 111 boxes in all the post offices.
not great if you get curious and try to see what’s in all the boxes!
The "give the locked box to the intern" attack is one of the most elegant things I've read. Nobody cracked the encryption. They just found a weaker colleague with the same vault access who was easier to convince. The GitHub finding though developers carefully scrubbing their chat logs before publishing, completely unaware the sensitive data was still sitting in the opaque blob they couldn't even read. That one stings.
“And what really makes me wonder is how the hell this even works across different labs' models...”
1000% Nelson…
and i keep going back to the idea that they are both heavily relying on LLMs for infrastructure.. so essentially it could be as simple as vibe coding. same pattern / distribution. They both needed a quick solution for fighting distillation attacks, and both landed on the same flaw.
The "locked box to the intern" framing nails the mechanism, but I'd push it one step further: encryption and access control got treated as the same guarantee here, and they aren't. The ciphertext held; nobody broke AES-GCM. What broke is the assumption that only the party holding the session key can meaningfully act on what's inside it. The provider's own infrastructure routed decryption to a weaker sibling model, so the real trust boundary was never the encryption, it was an internal routing decision nobody had audited as a security control.
That's a specific instance of a pattern that keeps showing up in agent-security postmortems: the boundary people build defenses around isn't the one that actually gets crossed.
that distinction is the interesting part imo. the trust boundary was way more about internal routing than crypto, which feels like the real tell. I like you push here. with crypto, it doesn’t take much to break trust.
It’s wild that they didn’t at least salt with user identity and model major version. But I suspect that their underlying infra depends on the ease of decryption and the portability makes for easier training, testing, instrumentation, and inspection. It’s not like they have an incentive to do the hard work to figure out how support end-to-end encryption. 🧐
right?! i was surprised to say the least. i’m guessing your right, it was “encrypted” to be as cheap and frictionless to decrypt. they may not have had dedicated cryptography people, or maybe rushed the process to stop distillation attacks…
Thanks for sharing this! It's really interesting how the researchers were able to get past the encryption. I get why big AI labs would do this to counter distillation, but it feels like they should’ve tested how well encrypting the chain-of-thought actually works more carefully.
absolutely agree. also, very interesting to see that this was a pattern, found it many of the labs. not a one off. a lot to read into that as well…
The not so encrypted chain of thought 💭😂
that’s right lol… 😂
I can't understand 😭
it’s a little tricky if you don’t have a cryptography background. but they essentially just misconfigured their backend. imagine the post office giving you a key that opens box 111. except it opens all the 111 boxes in all the post offices.
not great if you get curious and try to see what’s in all the boxes!
Umm I'm great full to you🥲
🙏🔥🙂
The "give the locked box to the intern" attack is one of the most elegant things I've read. Nobody cracked the encryption. They just found a weaker colleague with the same vault access who was easier to convince. The GitHub finding though developers carefully scrubbing their chat logs before publishing, completely unaware the sensitive data was still sitting in the opaque blob they couldn't even read. That one stings.
really clean way to frame it. not breaking the encryption, just socially routing around it is such a w… i love to see creative researchers.
This is nuts.
It shows the labs are going full speed on more capable models while putting below the bare minimum into security.
And what really makes me wonder is how the hell this even works across different labs' models...
“And what really makes me wonder is how the hell this even works across different labs' models...”
1000% Nelson…
and i keep going back to the idea that they are both heavily relying on LLMs for infrastructure.. so essentially it could be as simple as vibe coding. same pattern / distribution. They both needed a quick solution for fighting distillation attacks, and both landed on the same flaw.
hmm.
Yeah, that tracks. Same pressure, same tools, same shortcut...
The "locked box to the intern" framing nails the mechanism, but I'd push it one step further: encryption and access control got treated as the same guarantee here, and they aren't. The ciphertext held; nobody broke AES-GCM. What broke is the assumption that only the party holding the session key can meaningfully act on what's inside it. The provider's own infrastructure routed decryption to a weaker sibling model, so the real trust boundary was never the encryption, it was an internal routing decision nobody had audited as a security control.
That's a specific instance of a pattern that keeps showing up in agent-security postmortems: the boundary people build defenses around isn't the one that actually gets crossed.
that distinction is the interesting part imo. the trust boundary was way more about internal routing than crypto, which feels like the real tell. I like you push here. with crypto, it doesn’t take much to break trust.
It’s wild that they didn’t at least salt with user identity and model major version. But I suspect that their underlying infra depends on the ease of decryption and the portability makes for easier training, testing, instrumentation, and inspection. It’s not like they have an incentive to do the hard work to figure out how support end-to-end encryption. 🧐
right?! i was surprised to say the least. i’m guessing your right, it was “encrypted” to be as cheap and frictionless to decrypt. they may not have had dedicated cryptography people, or maybe rushed the process to stop distillation attacks…
Vibe coding FTW! 🤣
Thank you for this detailed and yet simple explanation!
What’s scary is the huge gap between perception of safety and security and real vulnerabilities in LLM architecture.
really appreciate it! yeah, this has been a pattern too. we see a lot of pretty basic controls slipping by.