Before we jump in, I’m excited to hand ToxSec over to a friend of mine, Mohib Ur Rehman, for the monthly guest post.
Mohib and the team at SK NEXUS spend a lot of time making complicated technology a little easier to understand, which makes him a pretty natural fit around here. Today he’s digging into data poisoning, malicious models, and the increasingly weird supply chain we’re building around AI.
So, I’ll get out of the way. Mohib, take it from here.
In our previous collaboration, we looked at AI prompt injections, one of the most discussed security risks around modern AI systems. While researching this topic, I recalled that I used to confuse prompt injection with data poisoning.
They sound similar because both involve manipulating AI systems, but they happen at completely different stages.
Prompt injection targets an AI system after it has already been deployed. Data poisoning happens during the training or development process, by corrupting the data or models that AI systems learn from.
The second one is harder to notice because the problem is often hidden before the system even starts interacting with users.
In February 2024, JFrog’s security research team scanned model files uploaded to Hugging Face, one of the largest public repositories for AI models. They found around 100 malicious models among millions of available files. Several of these models could execute arbitrary code when loaded by developers, despite existing security scanning measures.
This raised a broader question about how organizations handle the AI tools they bring into their systems.
Modern AI development depends heavily on external models, third-party datasets, and publicly available resources. When companies download a model or train systems using outside data, they also inherit risks that may not be visible at first.
This article looks at what data poisoning is, how attackers introduce it into AI systems, why it is difficult to detect, and what organizations can do to reduce the risk.
Let’s get started.
What Data Poisoning is
A model learns by studying examples. When trained on millions of text samples, it builds patterns that help it understand relationships between words, concepts, and ideas. These learned patterns shape how the model responds to future inputs.
Data poisoning targets this learning process before training is complete. An attacker adds carefully designed examples to the training data to influence the model’s behavior. Once the model learns from poisoned data, the changes become part of its internal parameters and remain after deployment.
Several distinct techniques fall under this category:
Backdoor Attacks
These types of attacks involve embedding a trigger in the training data for example a specific phrase, token, or pattern. The model learns to associate that trigger with a particular output. On any other input, it behaves normally. On the trigger, it does what the attacker intended, which mostly involves generating harmful content, bypassing a safety filter, or something else.
Label Flipping
These attacks alter the labels attached to training examples. For example, a fraud detection model trained on mislabeled data may learn to treat fraudulent transactions as legitimate. The model can still appear accurate during testing unless the evaluation data includes the manipulated examples.
Clean-Label Attacks
These attacks are harder to detect because the labels remain correct, but the examples themselves are designed to influence how the model learns. The data may look normal during review while still shifting the model’s behavior in a specific direction.
Across all three techniques, the goal is similar – a small number of poisoned examples can influence a much larger training dataset, creating targeted effects that remain after training and may not appear in standard evaluations.
How it Enters the Training Pipeline
Large models are not trained on hand-curated data. They are trained on datasets scraped from public web pages, code repositories and different document archives. The volume makes manual review impossible at any reasonable scale.
An attacker does not need to breach a lab’s infrastructure. They only need to place content in the right public location before it gets scraped. IBM X-Force’s Patrick Fussell described this to CSO Online:
“If we know the models are going to scrape Wikipedia every other week, all we have to do is be in that window. We can plant some bad data, and then we know that’s going to be ingested into the model.”
And the quantity required to poison the data is smaller than most teams assume. Research from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that injecting as few as 250 maliciously crafted documents can implant backdoors that activate under specific trigger phrases while leaving general model performance unchanged. That finding applied to specific experimental conditions and model architectures, but it demonstrates that poisoning does not require large scale data access to be effective.
Why Standard Testing Misses it
Model evaluation typically measures accuracy, coherence, and task performance on a benchmark dataset. A backdoored model can score normally across all of those.
Mithril Security demonstrated this in 2023 with its PoisonGPT project. The team modified a public GPT-J-6B model so it produced false historical information when asked specific questions. In other words, outside of those targeted prompts, the model continued to perform normally on standard benchmarks.
Standard benchmarks miss this class of attack because the attack is designed to survive them. Evaluating a model on general performance tells a firm whether the model is capable. But it does not tell them whether the model has been deliberately modified to behave in a specific, targeted way on specific inputs.
The Supply Chain Angle
A model reaches an enterprise through several stages, and each stage introduces a possible point of compromise:
Training data collection: Scraped or collected datasets can contain manipulated examples before training begins.
Pre-training: The model developer may unknowingly train on compromised data.
Public release: Open models can be modified, repackaged, or redistributed after release.
Fine-tuning: Additional training on community or third-party datasets can introduce new risks.
Distribution: Model repositories and cloud APIs create additional points where users must assess trust and provenance.
The JFrog finding mentioned above is one illustration of supply chain risk, though technically distinct from behavioral data poisoning. Those models carried malicious executable payloads in their file formats, not modifications through training data corruption. Both represent AI supply chain risk through different mechanisms. The OWASP LLM Top 10 covers both categories, ranking training data poisoning among the highest-impact risks for organizations deploying language models.
On the data side, datasets collected from public sources or third-party providers may contain manipulated examples before they are used for training. A single poisoned contribution in a shared dataset could affect multiple models built from that data.
AI Insider’s coverage of the Mercor supply chain incident was a great example of showcasing how security issues in widely used open-source AI tooling can spread into enterprise environments before organizations identify the source of the compromise.
Fine-Tuning Exposure
Fine-tuning is where many enterprises first interact directly with the model training process. An organization takes a pre-trained model and adapts it using internal data such as customer conversations, documents, support logs, or product manuals.
This creates two possible exposure points.
If the base model was already poisoned, fine-tuning may carry that risk forward. Adapting the model to a specific domain does not necessarily remove hidden behaviors embedded during earlier training.
The fine-tuning dataset itself can also become an attack surface. Data without proper access controls, provenance checks, or protection from unauthorized changes may introduce new risks before training begins. This is similar to software supply chain issues, where compromised dependencies can affect downstream systems.
RAG Pipelines and Inference-Time Risk
Retrieval-augmented generation (RAG) systems introduce a related but separate security risk. A RAG pipeline retrieves documents from a knowledge base at query time and passes them to the model as context.
Unlike data poisoning, this does not involve changing the model itself or its training data. Instead, attackers can place manipulated content in the retrieved documents and influence the model’s responses. This technique, known as indirect prompt injection, can affect model behavior without access to the underlying system.
Organizations using RAG systems should treat their document sources as part of the security boundary. Controls such as content provenance checks, access restrictions, and monitoring for malicious patterns can help reduce this risk.
What Organizations Can Do
Most of the practical controls here extend existing software supply chain security practices rather than requiring new programs from scratch.
Know Where Models Come From Before Using Them
Check for a published model card documenting training data, methods, and known limitations. Confirm whether the source repository provides integrity guarantees. Models without documented provenance carry risk that benchmark scores cannot reveal.
Apply Software Supply Chain Practices to Training Data
Training and fine-tuning pipelines need similar protections to software development pipelines. Access controls, audit logs, and integrity checks can help prevent unauthorized changes to datasets. A compromised training dataset can create a supply chain risk, similar to vulnerabilities introduced through compromised code dependencies.
Test for Targeted Behavior, Not Just General Performance
Red-teaming can help identify poisoned behavior by testing the model with sensitive topics, unusual prompts, and inputs outside its normal operating conditions. This approach focuses on understanding what the model should avoid, not only measuring what it can do.
Document the Model Supply Chain
A model bill of materials (AIBOM) tracks where a model came from, what data was used to train it, and how it has been modified over time. Without this record, organizations may struggle to identify where a model’s security risks were introduced.
Maintaining an AIBOM is necessary to help organizations identify where risks were introduced because it gives teams a clearer view of model provenance and helps them assess risks before deploying or updating AI systems.
Closing Thoughts
Data poisoning is a reminder that AI security starts much earlier than deployment.
A model can appear to perform well, pass benchmarks, and still contain hidden problems introduced during training.
The main challenge is that data poisoning does not always create obvious failures. Sometimes the model works exactly as expected until a specific condition triggers the behavior an attacker introduced which is why documentation, and testing are far more important than trying to fix problems after deployment.
Lastly, I want to thank ToxSec for giving me the opportunity to collaborate on this piece. AI security is a topic that deserves more attention, and I appreciate the chance to explore these fundamentals together.
And a big thanks to Mohib for joining us and putting this one together. Data poisoning is one of those AI security problems that gets much more interesting once you realize the attack can happen long before anyone ever types a prompt.
If you enjoyed Mohib’s work, you can find more from him on his Substack and over at SK NEXUS.
And as always, thanks for reading ToxSec. I’ll see you in the next one.
Great to see this collaboration!