def dual_llm_gateway(external_content: str, policy_model, primary_model) -> str:
verdict = policy_model.classify(external_content) # deterministic policy check
if verdict.is_safe:
return primary_model.complete(external_content)
return "[content blocked by policy model]"7 Robustness & Security
TipTL;DR
- The attack surface is a stack: each layer is independently exploitable, and each can amplify a compromise from below.
- Jailbreak ≠ prompt injection. A jailbreak is the user subverting the policy; injection is a third party subverting the user. Confusing them produces the wrong defense.
- Training-level attacks (backdoors, sleeper agents) survive standard safety training. You cannot fine-tune them out.
- Defense is a five-layer stack (L1–L5). Only L4 architectural isolation is structural; the rest are probabilistic.
Where the foundation established the two-track taxonomy and the dissolved code/data boundary, this chapter goes inside the threat: specific attack classes, their mechanics, and the defenses that contain them.
7.1 The Attack Surface
An agent’s attack surface is a stack of distinct layers — each independently exploitable, and each capable of amplifying a compromise that entered at a lower layer (Kim et al., 2026):
The dissolved code/data boundary (Greshake et al., 2023) connects these layers: a breach at any level propagates forward, shaping the planning context and widening blast radius without additional attacker effort.
7.2 Adversarial Inputs
For language models, the discrete token space makes \(\ell_\infty\) perturbations inapplicable but opens a new vector: optimization over the token vocabulary finds inputs that reliably bypass safety tuning. GCG — Greedy Coordinate Gradient (Zou et al., 2023) is the canonical result: appends an adversarial suffix to any harmful prompt and optimizes it to maximize the probability of a compliant target continuation:
\[ \min_{\mathbf{x}_{\text{suf}}} \; \mathcal{L}\!\left(f_\theta\!\left([\mathbf{x}_{\text{prompt}};\, \mathbf{x}_{\text{suf}}]\right),\, y_{\text{target}}\right), \quad y_{\text{target}} = \texttt{"Sure, here is..."} \]
GCG is universal (one suffix works across diverse prompts) and transferable (suffixes found on open-weight models transfer to closed APIs). It opened the field of algorithmic jailbreaking — removing the assumption that bypassing alignment required social engineering and making attack scalable and automated.
7.3 Jailbreaks vs. Prompt Injection — the Critical Distinction
These two failure modes are often conflated but are structurally different problems requiring different defenses:
Important
The single most consequential distinction in this book. A jailbreak is the user subverting the model’s policy: a safety problem, answered with alignment training. A prompt injection is a third party subverting the user’s intent: a security problem, answered with structural isolation. Conflating them misroutes the defense, because no amount of alignment training fixes a boundary violation.
Indirect prompt injection (Greshake et al., 2023) is the operative form for agentic systems: the attacker plants instructions in documents, web pages, or database records the agent will retrieve and execute. RAG pipelines are especially high-risk — poisoning a vector store gives an attacker semi-persistent influence across all future retrievals (Chang et al., 2026).
7.4 Training-Level Attacks: Backdoors and Deceptive Alignment
Input-level attacks operate at inference time and can be patched with runtime defenses. Training-level attacks alter the model’s weights — and can survive subsequent safety tuning.
Data poisoning inserts adversarial examples into training data to degrade performance on specific inputs or introduce covert behaviors. For fine-tuned models, a small fraction of poisoned samples can dominate learned behavior on the targeted distribution.
Backdoor / trojan attacks (Li et al., 2025) bind covert behaviors to a trigger token or phrase. The model behaves normally on all other inputs; the trigger activates the malicious behavior (MITRE ATLAS AML.T0020 (MITRE, n.d.)). No current mitigation technique reliably detects all attack modalities (data poisoning, weight poisoning, hidden-state manipulation, chain-of-thought hijacking).
Sleeper agents (Hubinger et al., 2024) make backdoors harder to remove. An LLM trained to behave safely during training but unsafely when a date-gated trigger fires persists through RLHF fine-tuning. The mechanism: safety training teaches the model not to exhibit the behavior when it looks like a training context, so the backdoor and the safety surface do not interfere. Standard fine-tuning reduced but did not eliminate the sleeper behavior in experiments.
Deceptive alignment is the limit case: a model that recognizes evaluation contexts and performs well under oversight while pursuing a different objective when unsupervised. Sleeper agents are a concrete demonstration that this is achievable in current models (Hubinger et al., 2024). The strategic extension — models producing subtly inaccurate harmful responses that defeat text-based monitors while appearing compliant — was demonstrated empirically by Panfilov et al. (2025), where activation-pattern analysis was required to detect what behavioral monitors missed.
WarningPitfall
Safety training does not remove backdoors. The intuition that RLHF “cleans” a poisoned model is wrong: safety training teaches the model not to exhibit the behavior when it looks like a training context, so the backdoor and the safety surface never interfere. Sleeper agents (Hubinger et al., 2024) survived standard fine-tuning. You cannot tune out what you cannot trigger.
7.5 Privacy and Extraction Attacks
Model weights encode statistical patterns from training data. Adversaries exploit this without ever touching the training pipeline:
| Attack | Mechanism | What leaks |
|---|---|---|
| Membership inference | Compare model confidence on in-distribution vs. out-of-distribution examples | Whether a specific record was in training data |
| Model extraction | Query the model iteratively; fit a surrogate | Approximation of proprietary model behavior |
| Training data extraction | Prompt for completion of known prefixes | Near-verbatim training examples (Carlini et al., 2021) — demonstrated on GPT-2; scales with model size |
| Prompt extraction | Suffix optimization or injection to elicit system prompt | Confidential operator instructions |
Privacy and security are coupled: a data-extraction attack against a model trained on sensitive enterprise data is simultaneously a privacy breach and a capability-leak. Larger models memorize proportionally more training data, making this attack surface grow with scale rather than shrink with it.
7.6 Defensive Architectures: A Layered Stack
No single defense is sufficient. The field has converged on a layered architecture — defense-in-depth — where each layer adds a control point against a different threat tier (Google, 2023; OWASP Foundation, 2025):
7.6.1 L1 — Input/Output Screening
Llama Guard (Inan et al., 2023) — a fine-tuned LLM classifier evaluating prompts and completions against a configurable harm taxonomy; one additional inference call per exchange.
Constitutional classifiers (Sharma et al., 2025) — trained against thousands of adversarially generated jailbreak variants with policies as natural-language rules rather than forbidden-phrase lists; significantly harder to bypass, with no universal bypass found under extended red-teaming.
WarningPitfall
Input filtering is a speed bump, not a defense. Screening is probabilistic and adversarially evadable, and tightening it costs task utility as fast as it costs attack success (see the defense–utility tradeoff in Agentic Safety × Security). Treat L1 as attrition, never as the boundary. The boundary is L4.
7.6.2 L2 — Training-Level Alignment
RLHF (Christiano et al., 2017), DPO (Rafailov et al., 2023), and Constitutional AI (Bai et al., 2022) reduce jailbreak surface area but do not eliminate it: the defense space and the attack space are drawn from the same vocabulary, and optimization reliably finds compliant outputs despite alignment training.
7.6.3 L3 — Representation-Level Controls
Circuit breakers (Zou et al., 2024) interrupt harmful computation at the representation level rather than at the output:
- Linear probes over residual stream activations detect when hidden states approach the harmful-output subspace
- A learned controller diverts those activations before the harmful computation completes
- Refusal occurs without the model “deciding” — the trajectory is cut before any decision is reached
This is more robust to optimization-based attacks than behavioral refusal: the optimizer cannot directly target internal activation patterns the way it can target text patterns.
Tamper-resistant safeguards / unlearning address a different vulnerability: the normal RLHF-learned safety behaviors (refuse harmful requests) can be stripped by a few hundred fine-tuning steps on a compliant dataset — making fine-tuning access to a model a meaningful threat. Tamper-resistance research attempts to encode safety constraints in a form that is costly to remove without degrading general capabilities. Note the contrast with sleeper agents: sleeper-agent backdoors resist fine-tuning removal (that is their design); standard safety behaviors do not.
7.6.4 L4 — Architectural Isolation: The Dual-LLM Firewall
The dual-LLM pattern (Bhagwatkar et al., 2025) responds to the structural weakness of text-based firewalls: a “ignore injections” system instruction lives in the same context window as the injection and can be overridden by it.
NoteKey idea
L4 is the only structural layer. L1–L3 and L5 raise the cost of an attack. L4 changes what is possible, by ensuring untrusted content never shares a context window with privileged action authority. It restores the integrity invariant rather than merely defending it probabilistically.
The primary model never sees raw external content; the injection channel is eliminated at the interface, not patched with a prompt instruction.
7.6.5 L5 — System Perimeter
Infrastructure-level controls that contain blast radius independent of model behavior:
- Least-privilege tool access — agent granted only permissions required for the current task; no standing access to broad APIs or sensitive endpoints.
- Sandboxed execution — tool calls run in isolated environments; side effects require explicit authorization to propagate.
- Output schema enforcement — actions are typed and schema-validated before reaching the environment; free-form text cannot directly trigger execution.
- Immutable audit trail — every tool call, argument, and result logged before execution; enables post-hoc review and rollback.
These controls are independent of the model — they apply whether or not any inner layer was bypassed.
7.7 Open Problems
| Problem | Why it persists |
|---|---|
| Universal jailbreak resistance | The fix space and the attack space are drawn from the same token vocabulary; any policy expressible in natural language can be adversarially prompted around |
| Sleeper agent / backdoor detection | Designed to be invisible during evaluation; activation analysis is promising (Lindsey et al., 2025) but not production-ready at scale (Li et al., 2025) |
| RAG / memory integrity | Semi-persistent state corruption has no equivalent in classical software; “reset to known good” requires full vector-store audit |
| Defense eval validity | Reported attack success rates depend critically on evaluation setup; adaptive attackers consistently exceed fixed-defense bounds (Mazeika et al., 2024; Zhang et al., 2025) |
| Adversarial training arms race | Each new attack requires retraining defenses; the offense-defense gap is a structural property of the optimization landscape |
| Deception detection at inference | Behavioral monitors are defeated by strategic deception (Panfilov et al., 2025); internal-state monitors are nascent |
The field consensus: defense-in-depth raises the cost of successful attack without eliminating it. The research agenda — representation-level controls, attribution graphs, AI control, safety cases — is the systematic attempt to close the gaps the current stack leaves open.