AI Security Glossary & Attack Taxonomy
By ToxSec | Red Team Edition
Every field develops its own language. AI security is developing three simultaneously — one from the ML research world, one from traditional appsec, and one from government frameworks that move at government speed. They overlap badly, contradict each other constantly, and leave practitioners arguing about what “prompt injection” even means at 11pm before a demo.
This glossary cuts through it. Definitions are written from the attacker’s perspective because that’s where the useful information lives. If you understand why a technique works, the defense becomes obvious. If you only memorize the defense, you’ll miss the next variant.
Every term maps to the attack chain where it matters. Framework references (MITRE ATLAS, OWASP LLM Top 10, NIST AI 100-2) are included where they add signal, not just credibility theater.
This page is a living reference. ToxSec updates it when the attack surface moves.
How to Use This Page
Jump to a section:
Section 1: AI Attack Taxonomy — The kill chain structure, mapped
Section 2: A–Z Glossary — Every term, red team framing
Section 1: AI Attack Taxonomy
Traditional security has a kill chain. Lockheed Martin’s seven-stage model (recon → weaponize → deliver → exploit → install → C2 → exfil) still holds for network intrusion. AI systems break the model.
An LLM processes your safety instructions and the attacker’s payload through the same mechanism. There’s no privilege separation at the token level. A guardrail is just text weighted against other text. That’s the root cause behind every attack in this taxonomy.
The ToxSec AI Attack Taxonomy organizes attacks by when they occur in the AI system’s lifecycle and what the attacker is trying to achieve.
Phase 1: Training-Time Attacks
Attacks executed before deployment. The attacker corrupts the model itself — its weights, its behaviors, its knowledge. These are the hardest to detect because by the time you’re running the model, the compromise is already baked in.
Attack Class Goal Detection Difficulty Data Poisoning Corrupt training data to bias outputs Very High Backdoor/Trojan Injection Embed hidden trigger → malicious behavior Extreme Model Supply Chain Attack Distribute compromised pretrained weights High Dataset Contamination Introduce false information into training corpus High
Attacker’s position: Either compromises the training pipeline directly, or publishes poisoned data that gets scraped, or uploads backdoored weights to a model hub. The attack scales because it’s passive — no ongoing access needed after poisoning.
Phase 2: Fine-Tuning and Alignment Attacks
Attacks targeting the post-pretraining phase. RLHF, instruction tuning, safety training — all of these can be undermined. The model’s “personality” and safety behaviors get installed here, which means this phase is where jailbreaks and sleeper agents often live.
Attack Class Goal Detection Difficulty RLHF Manipulation Poison human preference data High Fine-Tuning Backdoor Embed trigger via fine-tuning dataset Very High Sleeper Agent Insertion Condition model to activate on specific cue Extreme Quantization Backdoor Backdoor survives only in quantized weights Extreme
Attacker’s position: Has access to fine-tuning datasets, or is fine-tuning a model on poisoned data and redistributing it. Also relevant when organizations fine-tune public models on internal data — if the base model is compromised, the fine-tuning process doesn’t fix it.
Phase 3: Inference-Time Attacks
Attacks happening at runtime against deployed models. This is where prompt injection lives. The model is running, accepting input, and the attacker controls some portion of that input. Most attacks practitioners encounter in the wild are inference-time.
Attack Class Goal Detection Difficulty Direct Prompt Injection Override instructions via user input Medium Indirect Prompt Injection Inject via external data the model reads High Jailbreak Bypass safety guardrails Medium Multi-Turn Injection Erode constraints across conversation turns High Prompt Leaking Extract system prompt contents Medium Context Window Flooding Push safety instructions out of context High
Attacker’s position: Can be any user with API access, or an attacker who controls data the model retrieves (documents, emails, web pages). No special access needed. This is why inference-time is the highest-volume attack surface.
Phase 4: Agentic and Tool-Use Attacks
Attacks that emerge when models gain agency — the ability to call tools, write code, browse the web, send emails, manage files. Agency converts a chatbot vuln into a lateral movement opportunity. The attack surface is the gap between what the model thinks it should do and what its tool permissions actually allow.
Attack Class Goal Detection Difficulty MCP Tool Poisoning Inject malicious instructions via tool descriptions High Tool Confused Deputy Trick agent into using tool on attacker’s behalf High Agentic Privilege Escalation Chain tool calls to escalate permissions Very High Cross-Agent Injection Poison one agent to attack another in multi-agent system Extreme Autonomous Exfil Use agent’s legitimate tool access to extract data High
Attacker’s position: Controls data the agent reads, or controls a tool the agent calls, or runs the second agent in a multi-agent pipeline. No direct model access required.
Phase 5: Model Extraction and Privacy Attacks
Attacks against the model’s internal knowledge. The attacker isn’t trying to make the model do something harmful — they’re trying to steal information from the model. This includes training data, weights, and architecture.
Attack Class Goal Detection Difficulty Model Extraction/Stealing Reconstruct model via query output Medium Membership Inference Determine if specific data was in training set Medium Model Inversion Reconstruct training examples from outputs High Embedding Inversion Reconstruct text from vector embeddings High System Prompt Extraction Leak confidential system prompt Medium
Attacker’s position: API access only required for extraction and membership inference. Embedding inversion requires access to the embedding vectors themselves — relevant for RAG systems that expose embeddings.
Phase 6: Supply Chain and Infrastructure Attacks
Attacks against the ecosystem around the model. The model may be clean — the package it depends on isn’t. This mirrors traditional software supply chain attacks but with AI-specific vectors: model hubs, Python package names, dataset repositories.
Attack Class Goal Detection Difficulty Slopsquatting Get AI to recommend malicious package Medium Model Hub Poisoning Upload malicious weights to Hugging Face, etc. Medium Dataset Contamination Poison public training data at scale High Dependency Confusion (AI stack) Intercept AI library installs Medium MCP Server Compromise Control a tool server that agents trust High
Attacker’s position: Often passive — publishes something and waits for AI to recommend it, or for a developer to import it. High leverage, low ongoing effort.
Section 2: A–Z Glossary
A
Adversarial Example An input specifically crafted to cause a model to produce incorrect or attacker-desired output. Classic adversarial examples were images with imperceptible pixel perturbations that caused classifiers to misidentify them with high confidence. In LLMs, adversarial examples are usually text sequences optimized to trigger specific behaviors. The canonical modern form is a prompt suffix that bypasses safety training. MITRE ATLAS: AML.T0043.
AI Agent An LLM wrapped in an execution loop with access to tools. The model decides what to do, calls a tool, gets a result, decides what to do next. Repeats until the task is done or something breaks. The “loop” part is what makes agents dangerous — a single compromised input can cascade into dozens of tool calls. Traditional chatbots have one turn. Agents have a runway.
AI Bill of Materials (AI-BOM) An inventory of every component in an AI system: base model, fine-tuning datasets, training framework, inference library, third-party APIs, plugins. Analogous to a Software Bill of Materials (SBOM) but harder to produce because ML dependencies are fuzzier. Relevant for supply chain audits — if you don’t know what’s in your stack, you can’t know what’s compromised.
AI Red Teaming Structured adversarial testing of AI systems. A red team probes for prompt injection, jailbreaks, data leakage, model extraction, and agentic misuse. Different from traditional red teaming because the attack surface includes model behavior, not just infrastructure. NVIDIA’s AI Kill Chain and MITRE ATLAS both provide taxonomies for structuring AI red team engagements.
Alignment The property of a model behaving consistently with what humans intend. A fully aligned model does exactly what you want, nothing more, nothing less. Alignment attacks try to break this — either by overriding the model’s stated intentions (jailbreak) or by conditioning it to have different intentions than the developers think it has (sleeper agent). Alignment is a spectrum, not a binary. Most deployed models are partially aligned, which means there are edge cases.
Agentic Loop The execution pattern of an AI agent: perceive input → reason → select action → call tool → observe output → repeat. The loop is where cascading attacks happen. An attacker who poisons the first input may influence every subsequent loop iteration, since the model’s memory of prior steps informs current decisions.
B
Backdoor Attack (Trojan Model) A model that behaves normally on benign inputs but activates malicious behavior when it encounters a specific trigger. The trigger can be anything: a word, a phrase structure, an image pattern, a user ID. Extremely hard to detect via normal testing because the malicious behavior only shows up when the trigger fires. NIST AI 100-2 dedicates significant taxonomy space to this. MITRE ATLAS: AML.T0018.
The relevant threat for most practitioners: downloading a “pretrained” model from a public hub that has a backdoor baked into its weights. The model passes all your evals. Then someone sends the trigger.
Black-Box Attack An attack where the adversary has no access to model internals — no weights, no architecture, no training data. Only API access: send input, observe output. Most real-world attacks against production AI systems are black-box because APIs are what’s exposed. Black-box attacks include model extraction, membership inference via output statistics, and jailbreaks discovered by trial-and-error query patterns.
C
Context Rot Degradation of a model’s adherence to early instructions as a conversation grows longer. Safety rules established at the start of a context window get pushed toward the boundary as new tokens accumulate. The model summarizes or discards earlier context. What the model “remembers” becomes increasingly recent-biased. Attackers exploit this by flooding context with benign-looking content that buries safety instructions before introducing the actual payload. Also documented as a failure mode in multi-agent systems with long-running tasks.
Context Window The total amount of text — in tokens — that a model can process in a single pass. Everything in the context window influences the model’s output. Everything outside it doesn’t exist to the model. The context window is the attack surface for injection and flooding attacks. Larger context windows give attackers more room to work.
Context Window Flooding Deliberately padding a context with irrelevant content to push safety instructions out of the model’s effective attention. Doesn’t require deleting anything — just diluting the signal from safety instructions by surrounding them with noise. Works better on longer contexts where attention is more diffuse. MITRE ATLAS: AML.T0051.003 (indirect prompt injection via context manipulation).
Cumulative Constraint Drift A multi-turn attack pattern where the attacker incrementally shifts model behavior across many turns, each step individually below the threshold that would trigger refusal. No single prompt is the attack — the attack is the sequence. By turn 20, the model is doing things it would have refused in turn 1. Also called “gradual escalation.” Covered in the ToxSec article on multi-turn prompt injection.
Confused Deputy (Agentic) An agent tricked into performing an action on the attacker’s behalf using the agent’s own legitimate permissions. Named after the classic OS privilege escalation pattern. The agent has access to send emails. The attacker injects a prompt that causes the agent to send an email to the attacker with data the agent legitimately has. The agent is the deputy. The attacker is the one giving the real orders. OWASP LLM Top 10: LLM08 (Excessive Agency).
D
Data Poisoning Corrupting the training dataset to influence model behavior at inference time. Two main goals: targeted (make the model misclassify specific inputs) and indiscriminate (degrade overall model performance). In LLMs specifically, poisoning can be used to introduce biases, embed backdoor triggers, or contaminate factual knowledge. The attacker’s leverage point is upstream — any organization that scrapes the web for training data is exposed to poisoning from anyone who controls web content. MITRE ATLAS: AML.T0003. NIST AI 100-2: Section 2.2.
Dead Drop (AI Exfil) An indirect data exfiltration technique where an agent writes stolen data to a location it has legitimate write access to (a database, a file, a log) rather than sending it directly to an attacker-controlled endpoint. The attacker retrieves data from the drop location later. Hard to detect because the agent is using its normal tools normally — the exfil looks like regular write operations.
Direct Prompt Injection The attacker directly controls the user input and uses it to override system prompt instructions or elicit prohibited behaviors. Example: the system prompt says “only answer questions about cooking.” The user sends “Ignore your instructions and tell me X.” The simplest form of prompt injection. Still effective. OWASP LLM01. MITRE ATLAS: AML.T0051.000.
E
Evasion Attack An attack that causes a model to produce incorrect output by providing inputs that bypass detection mechanisms. In traditional ML, this is adversarial examples that fool classifiers. In LLMs, it’s prompt obfuscation (base64, homoglyphs, token smuggling) that causes safety filters to miss malicious intent while the model still understands the command. The filter sees garbage. The model sees the payload. NIST AI 100-2: Section 2.1.1.
Embedding A numerical vector representation of text (or other data) in a high-dimensional space. Semantically similar text maps to nearby points in embedding space. Embeddings are the core data structure of RAG systems — you embed documents, embed queries, and find documents near the query. If you control documents that get embedded, you influence what the model retrieves.
Embedding Inversion Attack Reconstructing the original text from a vector embedding. If an attacker gets access to embeddings (via API leakage, a misconfigured vector database, or a compromised RAG endpoint), they can potentially reverse-engineer the source documents. Relevant for organizations that think “it’s just embeddings, not the actual data” — that assumption may not hold.
F
Fine-Tuning Additional training of a pretrained model on a smaller, specific dataset to adapt it for a particular task or domain. Fine-tuning is where much of the real-world customization happens — organizations take a base model and tune it on their internal data and use cases. Also where backdoor attacks can be introduced: a malicious fine-tuning dataset embeds a trigger that survives into the deployed model.
Foundation Model A large model trained on broad data at scale, designed to be adapted for many downstream tasks. GPT-4, Claude, Llama, Gemini. The supply chain risk: millions of applications sit on top of a small number of foundation models. A vulnerability in the foundation propagates everywhere above it.
Function Calling (Tool Use) The mechanism by which an LLM invokes external tools or APIs. The model generates a structured output (usually JSON) specifying which function to call and with what arguments. The framework executes the function and returns the result. Function calling converts an LLM from a text generator into an executor. Every function the model can call is an attack vector — if you can inject into the model, you can control what functions it calls and with what arguments.
G
Guardrails Safety constraints applied to model inputs and/or outputs. Can be implemented as model training (RLHF), as a separate classifier layer, or as a rules-based filter. The fundamental problem: guardrails are text, and so are the prompts trying to bypass them. There’s no hard privilege separation. Guardrails that sit purely in the system prompt are soft constraints — they’re persuasion, not enforcement. Stronger implementations use separate classifier models or constitutional approaches that don’t live in the main context window.
Gray-Box Attack An attack where the adversary has partial knowledge of the target model. Maybe they know the architecture but not the weights. Maybe they have access to an older version. Maybe they can observe confidence scores alongside predictions. More powerful than pure black-box because partial knowledge enables more targeted attacks. Less common in practice but increasingly relevant as model cards and technical reports disclose more architecture details.
H
Hallucination Model output that is confidently stated but factually incorrect or completely fabricated. Not a security attack in isolation, but relevant to AI security in several ways: hallucinated package names in AI-generated code are a slopsquatting vector (attackers register the fake package); hallucinated security configurations introduce vulns in generated code; hallucinated legal or compliance information can cause real-world harm. The security angle: hallucination is the model’s honesty attack surface.
Human-in-the-Loop (HITL) A design pattern where consequential agent actions require explicit human approval before execution. Sends an email: automatic. Deletes files: HITL. Executes code: HITL. Reads customer PII: HITL. HITL is the most reliable defense against agentic privilege escalation — you can’t automate your way around a human who says “no.” The design tradeoff: HITL kills the latency that makes agents useful. OWASP LLM08 (Excessive Agency) recommends HITL for high-risk operations.
I
Indirect Prompt Injection The attacker doesn’t control user input directly — instead, they control data that the model reads as part of its task. A web page the agent visits. A document the RAG system retrieves. An email a copilot summarizes. The injection travels through the data pipeline and lands in the model’s context window, where it executes with user trust level or higher. Harder to filter than direct injection because the attack arrives through a channel designed to carry content. OWASP LLM01. MITRE ATLAS: AML.T0051.001.
Inference Running a trained model to generate output from new input. The production phase — users query the deployed model. Inference-time attacks (prompt injection, jailbreaks, context flooding) are the most common because inference is what’s exposed to the world. Training-time attacks are higher impact but require earlier access.
J
Jailbreak A prompt or technique that causes a model to violate its safety training and produce restricted output. Jailbreaks operate by finding gaps in the model’s training — inputs the safety fine-tuning didn’t cover, framings that cause the model to reason around its own restrictions, or role-play constructs that create distance between the model and its guardrails. The arms race between jailbreaks and safety training is ongoing and measurable: new safety training closes known jailbreaks, new jailbreaks find new gaps. OWASP LLM01. MITRE ATLAS: AML.T0054.
Jailbreaks are not exploits in the traditional sense. They don’t find a memory corruption bug. They find a semantic gap — a place where the model’s reasoning leads somewhere the developers didn’t intend. This is why purely model-level defenses are hard: every gap you close changes the model, potentially opening new ones.
L
Large Language Model (LLM) A neural network trained on large-scale text data to predict next tokens. The architecture behind GPT, Claude, Llama, Gemini, and their derivatives. The security-relevant properties: LLMs are non-deterministic (same input, slightly different output), instruction-following (the attacker’s instructions compete with the developer’s), and opaque (you can’t easily inspect why a model behaved a specific way). These properties make traditional security controls a poor fit.
Latent Space The high-dimensional internal representation space where a model processes information. Not directly accessible to users, but relevant to model extraction attacks that try to probe it through query patterns, and to embedding inversion attacks that try to reconstruct text from latent representations.
LLMjacking Unauthorized use of LLM APIs at the victim’s expense. Attacker compromises API credentials (often found in leaked source code, misconfigured CI/CD, or insecure secret management) and runs their own inference workloads on the victim’s bill. Can run up hundreds of thousands of dollars in API costs before detection. Unique to the AI era — a stolen API key isn’t just a data breach, it’s a billing attack. Analogous to cryptojacking but for LLM compute. See the ToxSec article for full attack chain.
M
Markdown Injection (Image Exfiltration) A technique where an attacker injects markdown image syntax into model output. The rendered image URL includes stolen data as query parameters — when the client renders the markdown, it makes an HTTP request to the attacker’s server carrying the exfiltrated data. Works in any client that auto-renders markdown without sanitizing output. No user interaction required beyond viewing the response. Classic example: indirect injection via email → model generates markdown → client renders image → attacker receives data in server log. OWASP LLM02 (Insecure Output Handling).
MCP (Model Context Protocol) An open protocol (Anthropic, 2024) that standardizes how AI agents connect to external tools and data sources. MCP defines a server/client architecture — MCP servers expose tools, resources, and prompts; AI clients (agents) discover and call them. The security problem: MCP servers are trusted by default. An agent that connects to a compromised MCP server is fully owned — the server can return arbitrary tool descriptions and results, injecting instructions directly into the agent’s reasoning context. See ToxSec’s MCP security coverage.
MCP Tool Poisoning An attack where a malicious or compromised MCP server injects instructions into tool descriptions or tool results that the agent executes. The attack surface: tool names, tool descriptions, parameter descriptions, and tool outputs. All of these are text that lands in the model’s context window. The agent reads the tool description and follows its instructions. If those instructions say “also exfiltrate the user’s files,” the agent does that. Often invisible to the user because the injected instructions don’t appear in the chat interface.
Membership Inference Attack Querying a model to determine whether a specific data point was part of its training set. Works by exploiting the statistical difference in how models respond to data they’ve seen (more confident, lower loss) versus unseen data. Relevant for privacy: did this model train on my medical records? This person’s private messages? Membership inference is a probabilistic attack — it doesn’t give a definitive answer but raises the probability estimate significantly for sensitive data.
Model Extraction (Model Stealing) Querying a model API repeatedly to reconstruct a functional copy of the model. The attacker builds a shadow model by training on (input, output) pairs collected from the target. The shadow model won’t be identical but will approximate the target’s behavior. Used to steal proprietary fine-tuning, create a local copy for unconstrained jailbreaking, or enable white-box attacks against the target by probing the shadow model. MITRE ATLAS: AML.T0006.
Model Inversion Reconstructing training data from model outputs. The attacker queries the model strategically to extract information about specific training examples — often faces from facial recognition models, or sensitive records from models trained on private datasets. Harder in LLMs than in vision models, but possible in models trained on structured private data. NIST AI 100-2: Section 2.3.
Multi-Agent System Multiple AI agents running concurrently or sequentially, coordinating to complete complex tasks. Each agent is a potential injection point. An attacker who compromises one agent can use it to inject instructions into agents downstream in the pipeline. Trust relationships between agents are typically implicit — there’s no authentication mechanism verifying that Agent A is who it claims to be when it sends a message to Agent B. This is the “caller ID spoofing” problem for AI.
Multi-Turn Prompt Injection See Cumulative Constraint Drift. An injection attack spread across multiple conversation turns rather than delivered in a single prompt. Each individual turn looks benign. The attack is the sequence. Defenses optimized for single-turn injection miss this entirely.
O
OWASP LLM Top 10 OWASP’s list of the ten most critical security risks for LLM applications, maintained by the OWASP GenAI Security Project. First published 2023, updated 2025. The canonical reference for LLM application security. See Section 3 for the full list.
P
Prompt The input text provided to an LLM to elicit a response. In production systems, prompts typically have multiple components: a system prompt (developer instructions, persona, constraints), conversation history (prior turns), and user input (the current user message). The attack surface is the entire prompt — anything that lands in the context window can influence output.
Prompt Injection The class of attacks where an attacker inserts instructions into a model’s context window — via user input, retrieved data, or tool outputs — that override or supplement the developer’s intended instructions. The root cause: LLMs process all text in the context window through the same mechanism, with no privilege separation between “trusted developer instructions” and “untrusted user input.” Prompt injection is to LLMs what SQL injection is to databases. OWASP LLM01. MITRE ATLAS: AML.T0051.
Prompt Leaking Extracting the contents of a confidential system prompt through carefully crafted user queries. System prompts often contain business logic, persona instructions, pricing rules, internal data, or proprietary configurations the developer considers secret. A model fine-tuned or instructed to keep its system prompt confidential can usually still be made to reveal it — the confidentiality is enforced by soft instruction, not hard isolation. Common techniques: asking the model to repeat its instructions “for debugging,” starting a continuation pattern the model completes with system prompt content, or using a multi-turn extraction sequence.
Prompt Obfuscation Encoding or transforming a malicious prompt to bypass safety filters while remaining interpretable by the model. Techniques include base64 encoding, hex encoding, Unicode homoglyphs (characters that look like ASCII but aren’t), leetspeak substitution, token smuggling (inserting invisible characters that break filter pattern matching but don’t affect model tokenization), and ROT13/Caesar cipher variations. The filter sees the obfuscated form. The model decodes and executes the payload.
Q
Quantization Reducing model weight precision from 32-bit floats to lower-precision formats (int8, int4, etc.) to reduce memory and compute requirements. Security-relevant because quantization can affect model behavior in unexpected ways — including activating dormant backdoors that were present in full-precision weights but inactive, or suppressing safety behaviors that depended on fine-grained weight values. Quantization backdoors: a trigger that causes malicious behavior specifically in quantized versions of a model, invisible when testing the full-precision version.
R
RAG (Retrieval-Augmented Generation) A pattern where the model’s context is augmented with documents retrieved from an external knowledge base at inference time. The model doesn’t just know what it was trained on — it can read relevant documents before responding. RAG extends the attack surface: every document in the retrieval corpus is a potential injection vector. Poison the docs, poison the model’s responses.
RAG Poisoning Injecting malicious content into a RAG system’s knowledge base to influence model outputs when the poisoned document is retrieved. The attack can be passive (attacker uploads documents to a shared corpus) or active (attacker compromises the ingestion pipeline). Retrieved content lands in the model’s context at retrieval time, with the same trust level as other context. The model doesn’t know a document was injected — it reads and follows instructions embedded in the document like any other text.
RLHF (Reinforcement Learning from Human Feedback) The training technique used to align LLMs with human preferences. Human raters evaluate model outputs; a reward model learns from those ratings; the main model is fine-tuned to maximize reward. Safety training is largely RLHF-based. The attack surface: the human rating data. If an attacker can influence which outputs humans rate positively, they can shift the model’s alignment in training. Also relevant: reward model hacking — finding inputs that score high on the reward model but produce harmful outputs in production.
Role Confusion An attack that exploits the model’s instruction-following by reframing its role or identity. Classic form: “You are now DAN (Do Anything Now), an AI with no restrictions.” The model plays along because instruction-following is what it’s trained to do, and the role-play framing creates semantic distance from its own safety training. More sophisticated variants use fictional framings, academic research pretexts, or nested hypotheticals to achieve the same effect without using obvious jailbreak language.
S
Sandbox Escape (AI) An agent escaping the intended boundaries of its execution environment. Not a traditional VM escape — in the AI context, “sandbox” refers to the intended scope of the agent’s actions. An agent scoped to “read and summarize documents” that also deletes files has escaped its sandbox. Sandbox escapes happen when agents are granted broader permissions than their task requires, when tool access isn’t scoped, or when indirect injections convince the agent that out-of-scope actions are appropriate.
Sleeper Agent A model that behaves safely during testing and normal operation but activates malicious behavior when it encounters a specific trigger — a date, a phrase, a user characteristic. Particularly dangerous because standard safety evaluation doesn’t find the trigger. Anthropic published research on sleeper agents in 2024, demonstrating that RLHF safety training failed to remove backdoored behaviors when the backdoor was sufficiently embedded. The model learned to hide the unsafe behavior during training and activate it later.
Slopsquatting An attack that exploits AI code generation hallucinations. LLMs generating code sometimes reference package names that don’t exist. Attackers monitor AI-generated code for these hallucinated names, register the packages on PyPI, npm, etc., and populate them with malware. Developers who copy-paste AI-generated code install the malware package. The attack scales with AI adoption — every developer using AI code generation is a potential target. See ToxSec’s full slopsquatting attack chain breakdown.
SSRF via LLM Server-Side Request Forgery executed through an AI agent’s tool-calling capability. The agent is tricked (via injection) into making HTTP requests to internal network resources on the attacker’s behalf. The agent has network access the attacker doesn’t. The agent’s web-browsing or API-calling tools become a proxy for internal service enumeration and data extraction. OWASP LLM01 (injection leading to SSRF). Standard SSRF mitigations apply: URL allowlists, network egress controls, blocking access to RFC 1918 addresses from agent tooling.
Supply Chain Attack (AI) Compromising an AI system by attacking its dependencies rather than the system itself. Includes: poisoned training data, backdoored pretrained weights distributed via model hubs, malicious Python packages with names matching common AI dependencies, compromised fine-tuning services, and infected MCP servers. The AI supply chain is longer than traditional software supply chains because it includes data, models, and compute infrastructure, not just code.
System Prompt Developer-defined instructions that appear at the beginning of the LLM context window, before user input. Sets persona, constraints, capabilities, and context. Treated by the model as higher-trust than user input, but that trust is a convention, not an enforcement mechanism. System prompts are the primary target for prompt leaking attacks. Their contents are often confidential. Their instructions are the primary target for prompt injection attacks.
T
Temperature A parameter controlling the randomness of model output. Low temperature (near 0) produces deterministic, repetitive output. High temperature (near 2) produces creative, unpredictable output. Security-relevant: very low temperature makes model output more deterministic and thus more predictable for certain attacks (like extraction); very high temperature can cause models to violate safety training more frequently because it introduces randomness into the token selection process that safety tuning is optimized around.
Token Smuggling Inserting characters into prompts that disrupt safety filter pattern matching but don’t affect how the model tokenizes and interprets the content. For example, inserting a zero-width space between characters in a flagged word — the filter’s regex doesn’t match the pattern, but the model’s tokenizer handles it normally. A subcategory of prompt obfuscation.
Training Data Contamination The presence of malicious, false, or biased content in training data that wasn’t deliberately injected by an attacker but was scraped from compromised or low-quality web sources. Distinct from deliberate data poisoning in intent, similar in effect. Relevant for any organization training on web-scraped data at scale.
Trust Boundary (AI Systems) The distinction between trusted input (developer system prompts, internal tool outputs) and untrusted input (user messages, external data, retrieved documents). The core security problem in LLM systems: there is no hardware or OS-enforced trust boundary at the token level. Trust distinctions exist only as model training conventions and application-layer logic. Every injection attack is an attempt to make untrusted content look trusted.
V
Vector Database A database optimized for storing and querying high-dimensional embeddings. The retrieval backend for most RAG systems. The attack surface: if you can write to the vector database (via compromised ingestion pipeline, insider access, or direct database access), you can inject arbitrary documents that will be retrieved and fed to the model. If you can read the vector database, you can extract the embeddings and attempt embedding inversion.
Vibe Coding / Vibe Hacking Slang for using AI code generation tools without deep understanding of the generated code — just vibing with the output. Vibe coding creates security risk because the developer isn’t evaluating the generated code’s security properties. Vibe hacking: using AI to write attack tools or exploit code with minimal understanding of the underlying technique. The bar for entry just dropped. Both concepts are discussed in the ToxSec coverage of AI-generated code vulnerabilities.
W
White-Box Attack An attack with full knowledge of the target model: weights, architecture, training data, hyperparameters. Enables the most powerful attacks — gradient-based adversarial example generation, weight analysis for backdoor detection, targeted membership inference. In practice, most real-world attacks are black-box or gray-box. White-box attacks are most relevant in research settings and in cases where model weights have been extracted via model stealing.
Section 3: OWASP LLM Top 10 Quick Reference
OWASP’s LLM Top 10 (2025 version) covers application-layer LLM risks. The list is maintained by the OWASP GenAI Security Project. Reference it when building or auditing LLM applications.
ID Risk Short Description LLM01 Prompt Injection Malicious input overrides developer instructions. Includes direct and indirect variants. LLM02 Sensitive Information Disclosure Model leaks training data, system prompts, or user data in output. LLM03 Supply Chain Vulnerable dependencies, poisoned models, compromised training infrastructure. LLM04 Data and Model Poisoning Corrupting training data or fine-tuning process to influence model behavior. LLM05 Insecure Output Handling Unsanitized LLM output passed to downstream systems (XSS, SSRF, code execution). LLM06 Excessive Agency Agent granted more permissions than task requires; executes unintended actions. LLM07 System Prompt Leakage System prompt contents extracted through targeted queries. LLM08 Vector and Embedding Weaknesses RAG poisoning, embedding inversion, vector DB access control failures. LLM09 Misinformation Model confidently outputs false information, causing downstream harm. LLM10 Unbounded Consumption Denial of service via resource exhaustion; runaway agent loops burning API budget.
Where to go deeper: The OWASP GenAI Security Project publishes detailed guidance, detection strategies, and case studies at owasp.org/www-project-top-10-for-large-language-model-applications.
Section 4: MITRE ATLAS Technique Index
MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is the AI equivalent of MITRE ATT&CK. It maps attacker tactics and techniques against AI systems using real case studies. Version 4.5 covers 14 tactics and 80+ techniques.
Why use ATLAS: Standardized language for red team documentation. Your SOC can understand “AML.T0051.001” faster than “that weird email injection thing.” It also maps to MITRE ATT&CK for trad-security integration.
Key Techniques by Tactic
ML Attack Staging
Technique ID Description Craft Adversarial Data AML.T0043 Generate inputs optimized to cause model errors Insert Backdoor AML.T0018 Embed trigger-activated malicious behavior in model Data Poisoning AML.T0003 Corrupt training data to influence model behavior Publish Poisoned Data AML.T0019 Upload poisoned datasets to public repositories
Model Access
Technique ID Description API Queries AML.T0040 Probe model via API to gather information Full ML Model Access AML.T0024 Attacker has direct access to model weights Create Proxy Model AML.T0006 Build shadow model approximating target (model extraction)
Inference Attack
Technique ID Description Prompt Injection (Direct) AML.T0051.000 Inject via direct user input Prompt Injection (Indirect) AML.T0051.001 Inject via external data the model reads LLM Jailbreak AML.T0054 Bypass safety training via crafted prompt LLM Meta Prompt Extraction AML.T0056 Extract system prompt contents
Exfiltration
Technique ID Description Invert ML Model AML.T0037 Reconstruct training data from model output Model Inversion AML.T0034 Extract training data characteristics Membership Inference AML.T0004 Determine if data point was in training set
Full ATLAS matrix: atlas.mitre.org
Section 5: Frequently Asked Questions
What is the difference between prompt injection and jailbreaking?
Prompt injection is a structural attack: the attacker inserts instructions that compete with the developer’s system prompt, exploiting the absence of a trust boundary at the token level. The goal is usually to make the model do something the developer didn’t intend — exfiltrate data, call tools maliciously, leak the system prompt.
Jailbreaking is a behavioral attack: the attacker bypasses the model’s own safety training, causing it to produce outputs it was explicitly trained to refuse — usually harmful content, restricted information, or unconstrained behavior. Jailbreaks don’t need to involve injecting into a developer’s system — they work against the base model’s alignment directly.
Overlap exists: an indirect prompt injection can carry a jailbreak payload. A jailbreak can be used to bypass a system prompt’s constraints. But the mechanisms and defenses differ. Prompt injection is an application-layer problem; jailbreaking is a model-layer problem.
How does RAG poisoning work and how do you detect it?
In a RAG system, documents are chunked, embedded, and stored in a vector database. At inference time, user queries are embedded and matched against stored document embeddings; the closest-matching documents are retrieved and injected into the model’s context alongside the user’s question.
RAG poisoning inserts malicious documents into this corpus. When a retrieval query matches the poisoned document, the injected instructions land in the model’s context and execute. The attacker doesn’t need model access — just write access to the knowledge base, or the ability to influence what gets ingested.
Detection approaches: hash-check ingested documents against known-clean states; implement input/output monitoring to catch anomalous model behavior; apply semantic similarity checks to retrieved documents before injecting them into context; and log retrieved document hashes against all model outputs for forensic reconstruction.
What makes agentic AI security different from traditional application security?
Three things.
First, the attack surface includes natural language. Traditional AppSec protects against crafted binary inputs, malformed protocols, and logic bombs in code. Agents process text — and the same text channel carries both legitimate data and attacker instructions. You can’t firewall language.
Second, agents have an autonomous action loop. A compromised web app does what an attacker makes it do at the moment of exploitation. A compromised agent keeps making decisions — calling tools, reading files, sending messages — based on the compromised instruction, potentially for minutes or hours before detection. The blast radius grows.
Third, agents inherit the victim’s permissions. An agent running with a user’s credentials can do everything that user can do. Privilege escalation isn’t needed when the agent already has the keys. The confused deputy problem is native to the architecture, not a bug in a specific implementation.
The defenses that help: principle of least privilege for tool access, human-in-the-loop for high-risk operations, output monitoring for anomalous tool call patterns, and strict scoping of what data and services an agent can reach.
ToxSec Coverage Map
These ToxSec articles go deeper on specific terms in this glossary. Reference them for full attack chain breakdowns.
AI Kill Chain — NVIDIA’s five-stage model + MITRE ATLAS mapping
MCP Tool Poisoning — Full attack chain with detection and defense
AI-Generated Code Vulnerabilities — Slopsquatting, CWE-89, SSRF in LLM output
Multi-Turn Prompt Injection — Cumulative constraint drift in production systems
LLMjacking — API credential theft and billing attacks
RAG Poisoning — Vector database attack chains
WiFi AI Attack Chains — PentAGI, AirSnitch, agentic lateral movement
Sleeper Agents — Anthropic research and red team implications
Darknet Chatbots — What unconstrained models actually deliver