Stealing an AI's Thoughts Without Breaking The Encryption

What if you could steal the hidden reasoning from one of the world’s most powerful AI models without cracking the encryption that protects it? That’s exactly what researchers did when they found a wonderfully strange workaround.

Feed the encrypted reasoning to a cheaper model and ask it to read the contents back to you… and it will!

Does The AI Actually Break the Encryption?

This week I was reading a paper called Stealing Reasoning Traces from Proprietary LLM APIs, and it really caught my attention because the attack sounds so completely ridiculous until you understand how APIs handle reasoning.

Read the research: Stealing Reasoning Traces from Proprietary LLM APIs Published on arXiv on August 10, 2026.

Models like Claude, GPT, and Gemini generate a bunch of internal reasoning before giving you their final answer. These days, due to distillation attacks, providers generally don’t want to hand all of that reasoning directly to the users because it can contain proprietary behavior, sensitive information and safety mechanisms.

It’s basically a picture of how the model arrived at its answer, and if you can get that information, you can train your own model off of it for a lot cheaper than it costs the frontier labs.

For example:

So what these labs started to do is have some of the APIs return the reasoning as an encrypted blob. You can’t read it. But your application holds onto it and sends it back during later API calls so the model can continue from its previous reasoning without the provider having to store the entire state server side.

And initially, this seems reasonable… except researchers discovered those encrypted blobs of text were surprisingly portable. That means they could move between conversations and between users.

They could even move between different models from the same provider.

And that last one is where things get fun.

Give the Locked Box to the Intern

Imagine Claude Opus does some complicated reasoning. You ask it a hard question, and it gets to work. The API doesn’t give you the plaintext reasoning, it gives you the encrypted reasoning block.

So you take the exact block and put it into a weaker model like Claude Haiku. Now, Haiku does not know the encryption string, and neither do you.

So how is it possible that it would be able to decode this?

Is it a hallucination?

No.

Haiku doesn’t need the key because the provider’s infrastructure actually recognizes the reasoning block, processes it, and makes the previous reasoning available to the current model as context. So now you have a much cheaper and generally easier-to-coerce model sitting there with the stronger model’s reasoning inside its context.

These faster models typically have less guardrails and aren’t as robust at defending themselves as the stronger models.

So researchers tell Haiku essentially to transcribe the reasoning it was given, and it does.

The researchers specifically selected the weakest compatible decoder they could find for each provider. Haiku 4.5 was used for Claude, GPT-5.6 Luna for GPT models, and Gemini Robotics 1.6 for Gemini.

And I really loved the absurdity of this attack because nobody actually broke the cryptography. But anybody familiar with cryptography can quite clearly see there were some really poor implementations here.

In my opinion, frontier labs should have much better practices around the integrity and confidentiality of this text. For example, this would’ve been completely bypassed if they had simply pinned encrypted text to the user or to the session.

But that’s a conversation for another time.

For this, essentially all they did was take a locked box from one employee over to another employee who had access to the same vault and was easier to trick.

---

---

Why the Weaker Model Matters

Now you might reasonably ask, why not just ask a powerful model to print its reasoning?

And again, that’s because the powerful model has protections specifically designed to stop it from doing that.

They will employ model-level safeguards saying “don’t reveal your hidden reasoning," and then they’ll also have system-level defenses looking for attempts to extract that. So whenever you attack the strongest model directly, that means you’re gonna be fighting uphill the entire process.

But for this attack, the encrypted format was compatible with weaker sibling models that didn’t necessarily resist extraction as effectively.

The researchers found that Haiku 4.5 could use one fixed extraction prompt across their Claude attacks. Interestingly, extracting through GPT-5.6 Luna was harder and required a lot more tricks, but it was the same basic idea.

The paper says Haiku extraction worked using one fixed prompt, while GPT-5.6 Luna required multiple prompt templates, repeated sampling, and other workarounds to deal with stronger anti-distillation safeguards.

The security boundary was around the encrypted data. But the model itself could still access what was inside, and the attacker could talk to the model.

That combination gets us to an uncomfortable state very quickly.

Then They Tried This at Scale

Because developers sometimes publish agent traces and API logs containing these opaque reasoning blocks.

They look encrypted, so they look harmless.

For this paper, researchers collected 6,708 publicly available agent trajectories from GitHub and Hugging Face and reconstructed several hundred thousand reasoning traces. Inside them, they found sensitive information which included API keys, passwords, access tokens, private keys, personal emails, names, and even postal addresses.

315,320

That’s how many encrypted reasoning blocks the researchers decoded from publicly available traces.

They reported 367 PII artifacts and 182 credentials across the collected data.

And across genuine user sessions, over 60 distinct API keys and 33 passwords were recovered.

Sometimes users had actually sanitized the visible conversation before publishing it, but the hidden reasoning still contained that information that no longer appeared in the visible chat.

So someone could look at the transcript, see the secret was removed, publish it, and unknowingly ship the secret anyway inside that opaque blob that they couldn’t read.

One weird detail: In genuine user sessions, the researchers found recovered sensitive artifacts that were completely absent from the visible chat history. They note these could have remained in hidden reasoning after visible text was scrubbed, or could have entered the reasoning through model memory.

Pretty awesome.

Encrypted Reasoning Can Also Carry Instructions

And if that wasn’t enough, somehow the paper got even weirder.

Researchers demonstrated that malicious instructions could live inside these hidden reasoning blocks. A victim could replay one of these blobs, the model could interpret the reasoning as part of its own thought process, and the malicious instructions would never appear in the visible conversation.

In one proof of concept, researchers created reasoning containing an instruction about uploading PowerPoint files, transferred that reasoning into GPT-5.6 Sol, then asked it for an unrelated script to edit a presentation, and the resulting script also uploaded the presentation to the attacker’s server.

This means that the vulnerability wasn’t only just stealing and being able to decrypt that text, but it’s also something that attackers might have been able to poison.

And nobody looking exclusively at plaintext conversations can see what’s happening inside. This research will end up being my next article, where I discuss chain-of-thought forgery, where prompt injection actually takes place through this process.

The Bigger Security Lesson

The architectural lesson here is much more interesting than the individual vulnerability.

Encrypted does not automatically mean untrusted parties cannot access the information.

You have to ask: who can submit the ciphertext? Where can they submit it? What identities, sessions, and models is the ciphertext bound to?

And in my opinion, most importantly, what system eventually gets permission to consume the plaintext?

The researchers’ proposed mitigations include binding reasoning state to the user and session. Their appendix describes a context-bound construction incorporating both user_id and session_id so reasoning cannot simply be moved between unrelated contexts.

In this case, researchers never needed the encryption key.

They already had something much more useful:

A model that did.

---

Thank you Andrei Savine, Apostolos Stamenos, MS, Jim Katzaman, M Hope, EL, and many others for tuning into my live video with Exploring ChatGPT!