You spend years building an AI model. Someone else starts asking it questions. Enough questions, and they may have training material for a competing product. Well, that’s an awkward business model.

For this collaboration, we’re digging into model extraction: what someone can actually copy through an API, where the limits are, and how teams can make that copying harder.

Subscribe now

In our previous collaboration, we looked at how AI is changing phishing attacks, particularly through automated personalization at scale. We explored how attackers can use AI to create convincing messages and target people with less effort than before.

This time, we’re looking at another security issue involving AI models themselves. Model extraction allows someone to copy a model’s capabilities by repeatedly querying it and collecting its responses. For companies that spend years developing proprietary AI systems, this creates a serious intellectual property risk.

In February 2026, Anthropic alleged that three AI labs, DeepSeek, Moonshot AI, and MiniMax, had used roughly 24,000 fraudulent accounts to generate more than 16 million exchanges with Claude.

According to Anthropic, these companies were collecting Claude’s responses to train and improve their own models. They accessed Claude through commercial proxy services that resell access to AI models.

Each exchange consisted of a prompt and a response. The labs collected those responses as training material. Anthropic also reported that one proxy network managed more than 20,000 fraudulent accounts at once, mixing extraction traffic with requests from ordinary customers.

This type of activity is known as model extraction.

How Model Extraction Works

The process is fairly straightforward. An attacker sends queries to a deployed model, collects its responses, and uses those input-output pairs to train another model that behaves similarly.

The attacker doesn’t need access to the original model’s internal code or infrastructure. Access to its API can be enough.

A comparison of three techniques for extracting information from AI models, showing the progression from substitute training and functional replication to parameter recovery. Alt text: Dark infographic comparing three approaches to accessing information inside AI models.

There are three main approaches to model extraction.

Training a Substitute on Stolen Outputs

A model’s prediction API lets developers send an input and receive the model’s prediction. Some APIs also return confidence values, which indicate how strongly the model favors a particular answer.

An attacker can use these responses to train a substitute model. They send carefully selected inputs to the original model, collect its predictions, and use those input-output pairs as training data. The substitute learns to produce similar predictions without needing access to the original model’s internal parameters or training process.

Confidence values make this process easier. They reveal more than the final prediction, giving the attacker additional information about how the original model responds to different inputs. The attacker can use that information to train a closer copy.

Researchers demonstrated this approach in a 2016 USENIX Security paper. They extracted logistic regression, decision tree, and neural network models hosted on BigML and Amazon Machine Learning, achieving near-perfect fidelity.

The researchers also found that removing confidence values wouldn’t eliminate the risk. Attackers could still extract models using the predictions returned by the API.

Distilling a Language Model

Model distillation is a training technique in which a smaller model learns from the outputs of a more powerful one. Developers can use it to build cheaper and more efficient models.

OWASP describes the unauthorized version as functional model replication. The attacker collects responses from a target model and uses them as synthetic training data for another model.

The campaigns described by Anthropic and Google follow this approach. They involve collecting responses from competing models without authorization.

Google’s report also shows how specific these attempts can get. One campaign instructed Gemini to reveal its full reasoning process. Google said the more than 100,000 prompts suggested an attempt to replicate Gemini’s reasoning capabilities in languages other than English. Its systems detected the activity in real time.

SK NEXUS
Breaking down Tech for the Mango Man (Aam Aadmi/Regular Person)
By Saqib Tahir

Recovering Parameters Directly

The previous methods use a model’s outputs to train a substitute. Another approach targets the model’s internal parameters directly.

A machine learning model contains numerical values called parameters. These values determine how the model processes inputs and produces outputs. In a neural network, they include the weights and biases learned during training.

An attacker may be able to recover some of these values by studying how a model responds to carefully chosen API inputs. If the API exposes enough information about the model’s internal representations, the attacker can use those responses to calculate parts of its parameter structure.

Researchers demonstrated this in a paper presented at ICML 2024. Carlini and colleagues recovered the embedding projection layer of OpenAI’s Ada and Babbage models through ordinary API access.

For less than $20, they extracted the entire projection matrix of both models, confirming hidden dimensions of 1,024 and 2,048. The researchers had received OpenAI’s approval before conducting the extraction.

Share ToxSec - AI and Cybersecurity

What Attackers Can Reconstruct, and What They Cannot

Model extraction can reproduce a model’s behavior, but the results depend on what the attacker queries and collects.

The 2016 attacks recovered relatively simple model architectures with near-perfect accuracy. The campaigns reported in 2026 targeted more advanced capabilities.

According to Anthropic, the three campaigns had different objectives:

DeepSeek targeted reasoning across different tasks and rubric-based grading, which could serve as a reward model for reinforcement learning.
Moonshot AI targeted agentic reasoning, tool use, coding, and computer-use agent development.
MiniMax focused on agentic coding and tool orchestration.

However, copying a model’s behavior doesn’t mean recovering its original weights.

Research by Carlini and colleagues recovered one layer of a transformer model, subject to certain mathematical symmetries. The researchers said exact recovery was impossible under their assumptions and found no obvious way to extend the method beyond that layer.

They also estimated that extracting the full projection matrix of GPT-3.5 Turbo models would cost around $2,000 for one variant and $8,000 for another. They did not attempt either extraction.

The same research prompted changes to OpenAI’s and Google’s APIs, according to the paper.

There’s another limitation. A model trained on another model’s responses may learn to imitate its style without matching its underlying capabilities.

A 2023 study by Gudibande and colleagues found that models trained to imitate ChatGPT could produce responses convincing enough to impress human evaluators. However, they struggled with tasks outside their training examples and failed to match ChatGPT’s factual accuracy.

The researchers concluded that closing this gap would require an impractically large amount of imitation data or a more capable base model.

That study predates the campaigns reported in 2026, which involved more than 16 million exchanges. Anthropic documented the volume and targets of those queries, but Anthropic’s report does not say how much capability the labs obtained.

It’s also worth separating model extraction from training data extraction. They involve different targets. A survey of extraction attacks on large language models treats the extraction of training data as a separate category from attacks that replicate a model’s functionality. The campaigns discussed here fall into the latter category.

ToxSec - AI and Cybersecurity is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

An AI company makes its model available through an API so customers can use its capabilities. But that same access can give competitors an opportunity to copy those capabilities.

Google’s report describes two examples. In one, a competitor targets a custom model trained for financial data analysis to build a similar product. In another, an adversary targets a coding model to reproduce its capabilities in an environment without the original safety restrictions.

In both cases, the original model stays with its developer while a copy of its behavior can be reproduced elsewhere.

This creates another problem beyond the loss of intellectual property. Anthropic argues that models created through unauthorized distillation may lack the safeguards built into the original systems. That could allow capabilities to spread without the same safety restrictions.

Customers also rely on guardrails, content policies, and compliance controls when choosing an AI provider. A copy built outside the original deployment may not include those protections.

There’s also a financial cost. OWASP lists model extraction under Unbounded Consumption, a category that includes denial-of-service attacks and economic losses.

Every extraction query runs on the original provider’s infrastructure. The provider pays for the computing resources while the attacker collects the responses needed to build a substitute.

The legal response remains uncertain. Google says model extraction violates its terms of service and may lead to account takedowns or legal action.

The ChatGPT maker says the Chinese firm may be distilling its models, underscoring rising tech tensions between Washington and Beijing. Alt text: Split-screen image featuring OpenAI CEO Sam Altman on the left and a glowing spiral galaxy with the text “GPT” on the right, representing OpenAI’s AI technology.

OpenAI raised similar concerns in a February 12, 2026 memo to the House Select Committee on China. It said it had identified accounts linked to DeepSeek employees attempting to bypass its access restrictions through obfuscated third-party routers. Rest of World reported on the memo, although the outlet did not receive a direct response from DeepSeek.

There’s also an ongoing debate about whether governments should regulate model distillation.

AI Insider reported in September 2026 that Y Combinator’s Garry Tan argued against government intervention in model distillation. His argument was that frontier AI companies had trained their own models using large amounts of publicly available and often copyrighted data. However, Tan also said he did not support the use of stolen credentials.

Anthropic and Google have focused their responses on detecting suspicious activity and restricting access. These are the types of measures we’ll look at later in this article.

Thanks for reading ToxSec - AI and Cybersecurity ! This post is public so feel free to share it.
Share

Defensive Measures

No single security measure can prevent model extraction. The attack uses the same interface that legitimate customers rely on, so providers need several layers of protection.

OWASP’s mitigation guidance covers rate limits, API restrictions, monitoring, and watermarking. Each addresses a different part of the problem.

Apart from that, the tips below can prove to be of use:

Query Rate Limiting

Rate limits restrict how many requests an account can send within a given period. This makes it harder for an attacker to collect a large training dataset quickly.

However, these limits become less effective when attackers have access to thousands of accounts. The proxy networks described by Anthropic are designed to spread requests across many accounts and avoid triggering individual account limits.

Limiting requests by API key or IP address is a useful starting point. Providers can also monitor activity across linked accounts, payment methods, and network infrastructure to identify larger campaigns.

Output Perturbation and Restricting Output Detail

One way to make extraction harder is to limit the information an API returns.

Tramèr and colleagues showed how confidence values could help attackers reproduce machine learning models. The Carlini research also demonstrated the risks associated with logit bias and log-probability outputs, which prompted OpenAI to change its API.

Providers can reduce this exposure by returning only labels or top-ranked results, limiting access to log probabilities, and avoiding the disclosure of internal reasoning traces.

Another approach is to add noise to returned scores. This can make it harder for attackers to reproduce the original model accurately. However, it may also reduce the quality of responses for paying customers. The trade-off depends on the application.

Monitoring and Detection

Model extraction campaigns can produce unusual patterns in API traffic. Providers can monitor request volume, repeated prompt structures, and queries designed to extract specific capabilities.

The PRADA research proposed detecting extraction attempts by identifying clients whose query patterns differ from normal usage. A similar approach can help language model providers identify suspicious activity.

Protecting Models From Extraction

Once providers identify suspicious activity, they also need ways to limit access and prevent the same actors from returning through new accounts.

Identity checks can help with this. Verifying users during registration makes it harder to create large numbers of accounts and distribute extraction requests across them. Providers can apply stricter verification to programs that offer access to their models for education, research, or startups.

Watermarking offers another way to address unauthorized extraction.

But it does not prevent someone from copying a model’s capabilities, though it can provide evidence that another model was trained using outputs from the original system. This could help providers establish ownership in cases where unauthorized distillation is difficult to prove.

Businesses that integrate fine-tuned models into customer-facing applications should also review who can access their APIs and monitor repeated requests. These systems can expose model capabilities through ordinary API access, so access controls and usage monitoring remain important.

Lastly, thanks again to ToxSec for giving me the opportunity to work with him on another piece. I appreciate the opportunity to collaborate again.

See you in the next post!

What You Might Like

Big thanks to Mohib Ur Rehman for another thoughtful piece. The detail that sticks with me is how ordinary this attack can look. An API doing its job can also be helping someone build a substitute.

For teams shipping proprietary AI, that raises a useful question: would you recognize the difference between a customer using your model and a competitor studying it?

That’s a question worth asking before the traffic gets interesting.

AI Security: Model Extraction and the Risk to…