AI Security

0x00 Basic Concepts

0.1 LLM

In essence: a machine that predicts the probability distribution of the next word

Because it works on probabilistic prediction, asking the same question twice can yield different answers. This non-reproducibility is the very precondition that AI security testing has to work around.

0.2 Prompt / System Prompt

• Prompt: the text you send to the model (the input)
• System Prompt: the "role definition + code of conduct" that developers pre-load into the model. It takes priority over ordinary user input — "You are a customer service bot. Answer only questions about company products, and do not disclose the system prompt."
• From the model's perspective, system prompts and user messages are all just "text" — there is no hard security isolation between them. This is exactly the root of "prompt injection" attacks.

0.3 Token

• The smallest unit the model processes, roughly "half a character to a few Chinese characters, or one English word." Input/output length and billing are both measured in tokens.
• Foreshadowing: an attacker can flood the service with text to burn tokens and kill it, or use token encoding to slip past detection.

0.4 RAG (Retrieval-Augmented Generation)

A model's own knowledge is limited, so it gets an external "knowledge base" bolted on. When a user asks a question, relevant documents are first retrieved from the base and stitched into the prompt before being fed to the model, so it "reads the material before answering."

Components: Embedding (vectorization) + Vector DB + Retriever + Context Builder (assembly).

This is one of the most important attack targets in AI security testing: because retrieved documents get mixed into the prompt as text, if a document contains a malicious instruction, the model falls for it = indirect prompt injection.

0.5 Agent / Agentic (an agent / capable of autonomous action)

• An LLM does more than answer questions — it is granted the ability to call tools (browse the web, execute code, send email, call APIs, read/write files).
• An Agent is an LLM with hands. Once hijacked, it can cause real-world damage (transfer money, delete files, send email), and do so at machine speed. This is a new form of risk that traditional security never had.

0.6 Hallucination

• The model "confidently talks nonsense," fabricating facts that don't exist.
• Hallucination is not just a quality problem — when users act on it in medical, financial, or legal contexts, hallucination becomes a security problem.

0.7 RLHF (Reinforcement Learning from Human Feedback)

• The key technique for training models to "follow instructions and behave": humans score the model's output, rewarding safe and useful responses and penalizing harmful ones.
• Foreshadowing: a model's safety "guardrails" are built mainly through this, and an attacker's objective is to find holes in those guardrails.

0.8 White-box / Black-box / Gray-box

• Describes how much an attacker knows about the model.
• White-box: access to the model architecture, weights, training data, and gradients (the deepest level of access).
• Black-box: only API access — you can send requests and observe responses.
• Gray-box: knowledge of the architecture / partial information, but no weights.
• In practice, most red-team scenarios simulate a black-box setting.

0x01 What Is AI Red Teaming

1.1 The Concept

AI red teaming = systematically "attack-testing" an AI system (model, training pipeline, API, integrations, deployment environment) through adversarial means to uncover vulnerabilities, anomalous behavior, and exploitable failure modes.

1.2 Why AI Systems Need Dedicated Adversarial Testing

1.2.1 Probabilistic Output

• Traditional software: same input → same output (deterministic).
• AI: submit the same prompt twice and the result may well differ; model temperature, sampling, and context all affect the output.
• • A single "successful jailbreak" doesn't mean it's reproducible (it may not work next time);
  • On the flip side, ten safe runs don't rule out a crash on the eleventh.
  • So you must do statistical testing (run many trials and look at the distribution) rather than a one-off verification. This is the biggest methodological difference between AI red teaming and traditional penetration testing.

1.2.2 Emergent Capabilities

• Models exhibit abilities that were never explicitly programmed, often surprising even their own creators.
• Example: GPT-4 can write working code, infer personal information from indirect context, and integrate knowledge across domains — abilities that "grew out of scaling up" rather than being engineered.
• Attackers will probe these emergent capabilities (for instance, piecing together your private details from casual chat), so red teams must explore them before attackers do.

1.2.3 An Attack Surface at the Semantic Level

• Traditional attacks happen at the binary/byte level (buffer overflows, byte manipulation, shellcode injection).
• AI attacks happen at the "semantic level" — the meaning of text, the structure of instructions, the context across turns of dialogue.
• This demands a completely different attacker mindset: attacking through natural language, narrative, impersonation, and psychological framing.

1.2.4 Complex Supply Chains

• Modern AI applications are assembled from a pile of components: pretrained base models, third-party APIs, vector databases, plugin ecosystems, and fine-tuning datasets of uncertain provenance.
• Every piece is a potential injection point.

1.2.5 Agent Autonomy

• Agents that can browse the web, write code, send email, and call APIs introduce a new category of risk: a compromised agent can cause real-world damage at machine speed, crossing every system it has been granted access to.
• The blast radius is no longer confined to data leakage — it can mean financial transactions, infrastructure changes, and persistent modification of other AI systems' memory or configuration.

1.3 Real-World Impact — The Chain of Evidence

These incidents are not confined to entertainment and chat: in medical diagnosis, financial decision-making, autonomous driving, critical-infrastructure monitoring, and military applications, the consequences of adversarial failure grow catastrophically.

1.4 Who Performs AI Red Teaming

• Internal security teams at model developers
• Security researchers at specialized firms
• Regulatory and standards bodies
• Enterprise security teams

0x02 Traditional Red Teaming vs. AI Red Teaming

Traditional and AI red teaming share the same underlying framework — adversarial thinking, controlled test environments, structured reporting — but they part ways when it comes to mechanisms, tools, mindset, and failure modes.

2.1 Comparison

The MSF, Burp, and Nmap you know are swapped for other tools in AI red teaming, and the framework shifts from OWASP Web to LLM — but the adversarial mindset itself transfers directly.

The biggest difference between traditional and AI red teaming is not technology, but mindset.

2.2 A Shift in Mindset

The mindset of a traditional penetration tester:

• Knows how software works "at the implementation level."
• Looks for the gap between "what the code is supposed to do" and "what it actually does when given unexpected input."
• That gap is usually well-defined and finite (there are only so many dead spots like buffer overflows).

The mindset of an AI red teamer:

• Think like an adversary who understands "how cognition works — or how a model learns to simulate cognition."
• An LLM doesn't process instructions through conditional logic; it computes probability distributions over text.
• You don't manipulate it by sending malformed packets. You do it by constructing narratives, shifting context, building personas, and exploiting the tension between the model's "instruction-following behavior" and its training objective (being helpful).

Conclusion: AI red teaming requires a hybrid skill set = classic security knowledge + machine learning theory + abilities approaching those of a social engineer / cognitive scientist.

Questions you need to be able to answer:

• What has the model learned?
• Which concepts does it associate with each other?
• What kind of "frame/narrative" can unlock behavior that direct instructions can't?
• Where do the safety properties produced by alignment training break down?

2.3 Where Traditional Security and AI Security Overlap

The good news: classic penetration testing experience transfers over in large part.

The toolbox got bigger, and the thinking evolved — it didn't get torn down and rebuilt.

0x03 The AI Attack Surface

Before attacking any system, a seasoned red teamer first enumerates the entire attack surface — every interface, component, and data path an adversary can influence.

The attack surface of an AI application is richer and harder to understand than that of traditional software.

3.1 Breaking Down the Top Ten Attack Surfaces

3.1.1 Model Weights

What they are: the numeric parameters that encode the model's learned behavior

How they're exploited:

• Adversarial fine-tuning — maliciously adjusting someone else's weights
• Weight extraction through repeated API queries
• Supply-chain attacks that swap in malicious weights (publishing a file that looks like the real deal)

Weights = the model's brain/memory. Stealing the brain or drugging the brain both count as attacks.

3.1.2 Training Data

What it is: the datasets used for pretraining, fine-tuning, and RLHF

How it's attacked: data poisoning — injecting adversarial examples or backdoor triggers into the data pipeline so the trained model behaves abnormally on specific inputs.

In plain terms: the AI's textbook gets laced with wrong information, so what it learns comes out crooked.

3.1.3 API and Inference Endpoints

What they are: the HTTP interfaces through which the model is queried

How they're attacked: authentication bypass, rate-limit bypass, parameter manipulation, and model inversion via systematic querying — reverse-engineering inputs/training data from outputs.

3.1.4 System Prompt

What it is: instructions the operator prepends ahead of the user message

How it's attacked:

• Prompt injection to extract its contents
• Context manipulation to bypass it
• If the prompt itself is assembled from data users can influence, it can also be poisoned.

In plain terms: the developer's instruction manual is also text, so it can be fooled, copied, and bypassed.

3.1.5 Output Processor

What it is: the code responsible for processing, rendering, and acting on LLM output

How it's attacked:

• LLM output rendered as HTML --> XSS
• Output drives shell execution --> command injection
• Output gets concatenated into SQL queries --> SQL injection

Model output is "high-entropy text" and must never be assumed safe by default; it can become a new attack channel.

3.1.6 Plugins and Tools

What they are: external capabilities granted to the AI Agent (browsing the web, executing code, email, APIs)

Concept: every tool is an executor — a hijacked tool-bearing Agent can send email, modify files, call APIs, and access external systems.

The Agent has grown arms and legs, and when those limbs fall into the wrong hands, things get dangerous.

3.1.7 Vector Database / RAG

What it is: the external knowledge base queried during RAG retrieval

How it's attacked: an attacker who can influence ingested documents can plant a prompt-injection payload, and once a legitimate query retrieves it, the payload executes.

The knowledge base itself can be booby-trapped — whoever queries it gets hit.

3.1.8 Orchestration Layer

What it is: frameworks like AutoGPT that chain LLM calls together, manage memory, and route between Agents.

How it's attacked: complex multi-Agent pipelines create muddled trust boundaries + recursive injection opportunities.

The more Agent layers you stack, the murkier it becomes whose word can be trusted between them — and attackers exploit exactly that gap.

3.1.9 Agent Memory

What it is: persistent memory storage across sessions.

How it's attacked: a single poisoned memory entry can influence that Agent's behavior across all future sessions and all users — creating a persistent, system-level backdoor.

Write one nasty line into the AI's long-term memory, and it'll never forget it afterward.

3.1.10 Deployment Infrastructure

What it is: cloud hosting, K8s clusters, GPU nodes, and model-serving infrastructure.

How it's attacked: traditional infrastructure attacks (container escape, credential theft, misconfigured IAM) all apply at this layer and can directly lead to the entire model being exfiltrated.

In essence, this is still "server security" — 100% classic penetration testing.

3.2 An Example Attack Chain

This attack chain touches six components: public page (ingestion surface) --> crawler/ingestion pipeline --> vector database --> LLM context window --> output processor --> browser frontend.

For traditional penetration testing, this is akin to web attacks where a single XSS point is harmless on its own, but becomes fatal when combined with CSRF + stored XSS + an administrator callback. The AI attack chain is an amplified form of that same chained-exploit mindset, spanning six heterogeneous components.

3.3 Trust Boundaries

Definition: A trust boundary is any interface where data crosses from a low-trust execution context into a high-trust one.

The traditional web trust boundary: user input is untrusted; database content is semi-trusted; server-side code runs with full trust.

AI introduces a new, deceptive trust boundary: the LLM context window.

• Everything inside the context window—system prompt, conversation history, retrieved documents, tool output, user messages—is processed by the same model through the same mechanism.
• The model is inherently unable to distinguish between the operator's instructions and the text in a document the user asked it to summarize.
• As long as an attacker can get text injected into the context window, they can potentially hijack the model's instruction-following behavior.

This is also the root cause of the prompt injection vulnerability class: it stems from an architectural-level limitation, not an implementation-level bug.

Traditional vulnerabilities are usually done once you patch them, but prompt injection is architectural—there is no way to ship a perfect patch. It can only be mitigated. This defines the defensive philosophy of AI security: always assume the model momentarily cannot tell which text is a trusted instruction.

0x04 MITRE ATLAS Framework

4.1 Overview

ATLAS (Adversarial Threat Landscape for AI Systems) is the most authoritative structured threat knowledge base in the AI/machine learning space. It is an AI-oriented version modeled after the classic network-attack MITRE ATT&CK framework, dedicated to cataloging tactics, techniques, and real-world case studies that specifically target ML environments.

Scale as of the end of 2025: 15 tactics, 66 techniques, 46 sub-techniques, 26 mitigations, and 33 documented case studies.

The framework is impossible to bypass:

1. It gives security communication a common language—when you say AML.T0051, the other side immediately knows which class of attack you mean.
2. It provides a structured foundation for threat modeling—work through the checklist and you're less likely to miss something.
3. It's a library of real attack patterns—every entry has been validated against a real incident, not dreamed up on a whim.
4. From a practical standpoint, it aligns exactly with how enterprise security teams think about threats—report in ATLAS terminology and the defender can pick it up instantly.

Tactics = the high-level objectives an attacker holds at each stage of an attack campaign (what they want to accomplish at that stage).

4.2 The Sixteen Core Tactics

AML.TA0001 Reconnaissance

• Collecting information about the target AI system: searching public research material to understand the architecture, querying public APIs to infer the model's version and capabilities, scouring GitHub for leaked training scripts/config files, and digging through documentation for tool integrations.
• Example: crafting inputs designed to trigger API errors that leak version/config details in the error messages; systematically querying to estimate the context-window size and map out memory behavior.
• In traditional security terms, this is the equivalent of information gathering.

AML.TA0002 Resource Development

• Acquiring, developing, or repurposing the capabilities needed for an attack. The most typical example is training or obtaining a proxy model—a local copy or approximation of the target model used to develop attacks offline.
• It also includes: standing up infrastructure to host malicious datasets, generating adversarial examples via gradient-based methods, and buying cloud GPU compute for high-throughput model extraction.
• You build your ammunition before the attack, the same way traditional pentesting first sets up tools, C2, and platforms.

AML.TA0003 AI Attack Adaptation

Traditional pentesting is like using a master key (SQL injection, buffer overflow — just try it anywhere). But a master key doesn't open AI systems — each AI's brain is wired differently.

• AI Attack Adaptation is the attacker building a custom mold for one specific AI — deliberately developing a bespoke weapon.
• Writing scripts, tuning parameters, crafting disguises
• If you don't adapt the attack to the specific AI model, you'll struggle to get past its rate limiting.

AML.TA0004 Initial Access

• Securing a first foothold within the AI ecosystem. There are many paths: exploiting ML supply-chain vulnerabilities (installing a tampered-with popular ML library); physically accessing inference infrastructure or compromising a victim with legitimate model access; abusing unauthenticated, insecure APIs; and prompt injection, when it serves as the entry point for a more complex attack, also falls in this category.
• Initial access to AI systems is often easier than to traditional systems—because public-facing LLM applications are inherently designed to accept natural language from any user.
• It's exactly what we covered earlier: the entry point/foothold, but with a much lower barrier to entry.

AML.TA0005 AI Model Access

• Covers the various ways an attacker obtains "meaningful query access" to the target model.
• • White-box: full access, usually via supply-chain compromise;
  • black-box: API only;
  • gray-box: API plus knowledge of the architecture or training data.
• Impact: the level of access directly determines which techniques become available downstream—gradient-based adversarial examples require white-box access, while model extraction and black-box jailbreaking only need inference capability.
• Which access level your exercise simulates must be decided up front as an explicit boundary.

AML.TA0006 Execution

• Making the target run malicious code or perform adversarial behavior. In the ML context this takes concrete forms such as: triggering a backdoor planted during training (e.g., a certain 4x4 pixel pattern appearing in the corner of an image → always classified as "benign"); using the LLM's code-execution tools to run attacker-controlled code; or leveraging the Agent's browsing capability to jump to malicious pages containing additional injection payloads.
• In AI systems, "execution" is often achieved through the model's own intended, expected behavior, rather than by exploiting a code vulnerability.
• In traditional execution, the code is one you wrote; in AI execution, you're tricking the model into doing the work for you.

AML.TA0007 Persistence (persistence)

• Establishing continued access or influence that survives sessions, updates, and detection events. AI persistence is exceptionally strong—it can act on the model's knowledge layer or memory layer:
• • Poison the fine-tuning data -> backdoors persist even in re-trained versions
  • Inject into the cross-session memory store -> an Agent's behavior in all future sessions is permanently altered
  • Plant data in a shared knowledge base (RAG store) -> it fires the moment a related query retrieves it.
• In 2025, ATLAS added two new persistence vectors: Agent memory manipulation and AI Agent configuration modification.
• Traditional persistence relies on "writing startup scripts / dropping a webshell"; AI persistence relies on "poisoning its memory and textbooks."

AML.TA0008 Privilege Escalation

• Expanding the scope of access or influence within an AI system.
• Agent scenario: persuading a low-privilege Agent to invoke a high-privilege tool; manipulating the orchestration layer to grant an Agent capabilities beyond what was intended; exploiting the confused deputy vulnerability—where one Agent borrows another Agent's credentials.
• LLM scenario: the attacker escalates from being "a user with prompt access" to "bypassing the system prompt," from which point they can hand the model author-level instructions.
• Privilege escalation in a new disguise

AML.TA0009 Defense Evasion

• Evading detection or bypassing security mitigations. The ML context offers a wealth of examples:
• • Adversarial perturbations: altering inputs in ways imperceptible to humans so AI security detectors misclassify them;
  • Jailbreak prompts: bypassing input filters to achieve the same result as a directly harmful request;
  • Token obfuscation / character substitution (using Unicode look-alike characters), Base64-encoding harmful instructions, multi-hop prompt chains;
  • In Agentic systems, using a "benign-sounding cover story" to make monitoring systems classify injected instructions as legitimate data.
• The encoding, obfuscation, and fragmentation tricks used to bypass WAF/antivirus can be reused in AI as well.

AML.TA0010 Credential Access

In a traditional pentest, getting to the data means escalating privileges, bypassing ACLs, and brute-forcing — noisy and easy to detect. But an AI system's access control rests on assorted string credentials — get the right string and you're in.

• Prompt injection
• Error-message disclosure: persuade the AI to leak the information by accident.
• Man-in-the-middle interception: the credential traffic isn't going over HTTPS, or certificate validation is lax.
• Digging through public repos: searching for stale config.json or .env files left behind.
• Supply-chain credential hijacking: if the AI depends on third-party services (e.g., cloud storage, vector databases), work backward from those third-party credentials to break into the AI.

AML.TA0011 Discovery

• Once inside the ecosystem, the attacker maps out the architecture: figuring out which model is in use, where sensitive data sits, what tools an Agent has, which other systems it's connected to, and how data flows.
• Techniques: probing the model to extract the system prompt; using an Agent's discovery tools to enumerate APIs; crafting queries to reverse-engineer RAG store contents; identifying the activation triggers that cause an Agent to run specific workflows.

Analogy: internal network reconnaissance / asset inventory before lateral movement.

AML.TA0012 Lateral Movement

• Moving from a compromised component to adjacent systems.
• • AI ecosystem examples: using a compromised customer-facing LLM to poison a shared knowledge base (which internal enterprise Agents also use); leveraging an Agent's tool access to reach downstream systems (databases, code repositories, communication platforms).
• Multi-Agent architectures are especially fragile: every Agent is a potential stepping stone, and prompt injection can be designed to propagate through Agent-to-Agent communication channels.
• It's the AI version of lateral movement + internal phishing—but instead of "human-to-human," it's "Agent-to-Agent."

AML.TA0013 Collection

• Harvesting valuable data. Primary targets: training data extraction (systematically querying to recover samples the model memorized); system prompt exfiltration (pulling out operator instructions containing API keys, business logic, and sensitive configuration); user conversation history; vector database contents.
• During the collection phase, the target shifts to "the data inside the model's head and inside its context."
• Recorded in ATLAS: RAG database probing and inference API exfiltration.

AML.TA0014 Command And Control

• Establish a covert "remote-control channel" inside the AI system — so that an implanted backdoor, poisoned weights, or a hijacked agent can receive instructions from an external attacker and exfiltrate the stolen data.
• Techniques include: - Hiding instructions inside seemingly innocuous user queries (e.g., "what's the weather today") that actually act as backdoor trigger phrases. - Abusing the AI's web-search capability for data exfiltration — having the AI stitch sensitive content into a normal-looking "summary" and send it to an attacker-controlled URL. - Even encoding stolen data as token probability distributions and dripping it out bit by bit through API response headers.

AML.TA0015 Exfiltration

• Getting data out of the AI system. Semantically similar to ATT&CK exfiltration, but AI has a twist: data can be exfiltrated through the model's own output—hidden in replies, embedded in generated images, concealed in suggested hyperlinks, or smuggled through API response metadata.
• LLM exfiltration is particularly stealthy: the model's output is high-volume and highly variable, and without AI-assisted detection it's very hard to monitor covert data channels.
• Analogy to traditional pentesting: DNS tunneling and steganography—only now the carrier is LLM output.

AML.TA0016 Impact

• Achieving the ultimate goal: destroying, degrading, or weaponizing the AI system.
• Techniques: model DoS (flooding it with high-resource-consumption queries); data manipulation (making the AI system corrupt or delete data); generating and spreading disinformation at scale; deleting or tampering with training data; weaponizing an Agent to execute unauthorized real-world actions (transfers, communications, configuration changes).
• The most severe scenario: an Agentic AI with broad tool access and no human oversight.
• Impact = the final blow of destruction, but with "manipulating model output" added on top of traditional Impact.