def is_in_scope(instruction: str, action: dict) -> bool:
return True # deployment-specific: semantic similarity, rule engine, or second model
def pre_action_gate(original_instruction: str, planned_action: dict,
irreversible_verbs: set) -> str:
verb = planned_action.get("tool", "")
if verb in irreversible_verbs:
return "HOLD — irreversible action requires human confirmation"
if not is_in_scope(original_instruction, planned_action):
return "BLOCK — action outside original task scope"
return "PASS"
irreversible = {"send_email", "delete_file", "execute_code", "transfer_funds"}12 Agentic Safety × Security
TipTL;DR
- An agentic system introduces failure modes that do not exist in any of its components in isolation. A correct model, a correct tool, and a correct retrieval pipeline can compose into an unsafe agent.
- Agentic failures are trajectory properties. They emerge across steps, so single-turn evaluation cannot see them.
- Text firewalls cannot fix injection. A model cannot be both executor and trusted auditor of its own context.
- The fix is structural: separate the data channel from the control channel, restoring the integrity invariant.
- Defense is layered across the loop (perceive → plan → act → memory), and every layer trades safety against utility.
Every preceding chapter converges here. The foundation established why safety and security split into two tracks — and why agentic systems are the seam where they reunite. Robustness & Security catalogued the attack classes and the defense layers. Evaluation showed how to measure what survives. This chapter closes the loop: how the failures compound when a model runs inside an action loop, why each compounding makes it worse, and what a defensible agentic architecture actually looks like end-to-end.
The organizing claim: an agentic system introduces failure modes that do not exist in any of its components in isolation. A correct model, a correct tool, and a correct retrieval pipeline can compose into an unsafe agent. The threat is not in the parts — it is in the connections.
12.1 1. The Agentic Attack Surface
A deployed agent is not a single model — it is a loop that connects a model to memory, tools, other agents, and an environment. Each connection is a capability and an attack surface. Kim et al. (2026) formalizes this as a taxonomy of vulnerability classes spanning the full agent stack:
Two properties of this surface distinguish it from a single-model system:
Depth — a compromised entry at one layer propagates forward through the loop. Poisoned retrieval content becomes trusted planning context; a compromised tool output shapes the next action; a memory write persists across sessions.
Amplification — each step the agent takes based on a compromised input can authorize further actions, expanding the blast radius without additional attacker effort.
Important
Component-wise review cannot find these failures. Every row above is a channel that exists because the agent is useful, so auditing the model, the tools, and the retrieval pipeline separately will clear all three and still miss the vulnerability. The unit of security analysis for an agent is the loop, not the component.
The dissolved code/data boundary is the root cause in all six rows: the same channel that carries legitimate content can carry an instruction.
12.2 2. Single-Agent Risk: When the Loop Is the Threat Surface
Static safety benchmarks miss agentic failures because they evaluate a model at a single turn. Agentic failures are trajectory properties — they emerge across steps, not at any individual step.
12.2.1 2.1 Accumulated context poisoning
An agent’s planning context at step \(t\) is a concatenation of everything it has perceived, decided, and observed so far:
\[ c_t = [i_0,\; o_1, a_1, r_1,\; o_2, a_2, r_2,\; \ldots,\; o_t] \]
where \(i_0\) is the original instruction, \(o_t\) the observation, \(a_t\) the action, and \(r_t\) the tool result. If any \(r_k\) (a tool result fetched from an untrusted source) contains an injected instruction, it is now embedded in \(c_t\) alongside the trusted instruction — and later planning steps treat the entire context as input. The earlier the injection, the more planning context it has already shaped.
This is structurally different from single-turn injection: the attacker does not need the model to comply now — they need it to comply later, after the injected content has been incorporated into the agent’s working state.
12.2.2 2.2 Tool-use privilege escalation
The danger of tools is not that they exist — it is the step sequence they enable:
An injection arriving at search_web() — the second step — can steer the output of send_email() and execute_code() without ever touching the original instruction. The agent escalates its own privileges by chaining tools; the attacker only needed to control one step.
12.2.3 2.3 Irreversibility
Many agent actions have no undo: emails sent, files deleted, API calls that transfer funds, database writes. A model that hedges on a text output (it can be discarded) faces a fundamentally different risk calculus than a model that pulls a trigger in the world. The key asymmetry:
- Cost of over-caution — a task not completed; user inconvenienced.
- Cost of under-caution — an irreversible action; cannot be walked back.
Current alignment training does not reliably teach models this asymmetry. Helpfulness optimization provides strong signal for task completion and penalizes unnecessary refusals, but no equivalent training signal encodes the cost of irreversible side effects (Amodei et al., 2016). The result: models under-weight caution at precisely the moments when under-caution is most costly.
WarningPitfall
There is no rollback for the real world. Safety training optimizes a symmetric-looking tradeoff (refuse too much vs. comply too much), but the world is not symmetric: an unhelpful answer costs a retry, a sent email costs everything after it. Guardrails must sit before the irreversible action. A monitor placed after it is not a control at all.
12.2.4 2.4 Oversight decay with horizon length
Human oversight degrades as task length grows: the operator who specified the original instruction is not watching step 47 of a 50-step agentic task. The model’s behavior at late steps is governed primarily by in-context planning — shaped by accumulated tool results, not the original instruction. This is the oversight gap that scalable oversight research attempts to close (Bowman et al., 2022; Engels et al., 2025).
12.3 3. Prompt Injection in the Agentic Loop
Prompt injection in an agentic setting differs from the static case in three ways: it persists, it propagates, and it amplifies. Greshake et al. (2023) established the base case; the agentic loop gives attackers compounding leverage.
12.3.1 3.1 Injection entry points across the loop
| Entry point | Attack type | Why it compounds |
|---|---|---|
| ① RAG / retrieval | Environmental injection — attacker plants instructions in documents the agent will fetch | Content enters the planning context before any output screening sees it |
| ② Tool output | Tool response hijacking — a compromised API or MCP server returns crafted content | Appears with the trust of a tool result, not a user input |
| ③ Memory / state | Stored injection — malicious content written to persistent memory in a prior session | Reactivates on every future session; attacker is no longer online |
| ④ Inter-agent message | Cross-agent injection — a compromised agent passes an instruction to a peer | Inherits the peer’s trust level and bypasses per-agent input filters |
12.3.2 3.2 Why runtime text firewalls are insufficient
The naive defense is a text filter — screen every input for known injection patterns. Bhagwatkar et al. (2025) demonstrate why this fails structurally:
- The primary model receives a prompt containing both trusted instructions and retrieved content in the same context window
- A text-based firewall operating on that same context is itself subject to the injection it is meant to block
- Sufficiently crafted content can override the filter because filter and payload share the same natural-language context; the model cannot distinguish which part has authority
The underlying issue: a model cannot be both the executor and the trusted auditor of its own inputs when both live in the same context window.
WarningPitfall
“Just tell the model to ignore injections” is not a defense. It is the field’s most common mistake. The instruction to ignore malicious content and the malicious content itself occupy the same context window, with no channel marking which one holds authority, so a sufficiently crafted payload overrides the guard. A text firewall is subject to the very attack it exists to block.
12.3.3 3.3 The data/control separation principle
The architectural fix is structural, not linguistic: separate the data channel (content the model may reference) from the control channel (instructions the model may execute). The interface-isolation principle states:
Untrusted content may shape what the agent says or retrieves, but must not determine what actions it takes.
Formalizing from the security foundations integrity invariant:
\[ \forall\, u, u' :\quad \texttt{action}\!\left(\pi(i,\, d_u)\right) = \texttt{action}\!\left(\pi(i,\, d_{u'})\right) \]
where \(i\) is the trusted operator instruction and \(d_u\) contains arbitrary untrusted content. An action that varies with \(u\) is a violation — a successful injection. Defenses are mechanisms that restore this invariant.
NoteKey idea
The fix is structural, not linguistic. Injection is not solved by better wording, better filters, or a more obedient model. It is solved by ensuring untrusted content cannot reach the channel that authorizes actions. Every guardrail in the next section implements one principle: untrusted content may shape what the agent says; it must never determine what the agent does.
12.4 4. Guardrail Architectures for Agentic Systems
The five-layer defense stack from the robustness chapter was designed for static models. The agentic loop requires each layer to be placed and coordinated differently — and adds two layers that do not exist in the static case: pre-action validation and mid-loop interruption.
12.4.1 4.1 G1 — Input screening at the perceive step
Every external input — user message, retrieved document, tool response, inter-agent message — passes through a classifier before entering the planning context. Constitutional classifiers (Sharma et al., 2025) are the current state of the art: trained against thousands of adversarially generated attack variants, with policies stated as natural language rules. Llama Guard (Inan et al., 2023) provides a configurable harm taxonomy at lower compute cost for less adversarially demanding inputs.
Agentic wrinkle: the volume of external inputs is far higher than in static deployments:
- Every retrieval result and tool response is an untrusted external input requiring screening
- One additional inference per input is costly at scale — a 100-step task becomes 200 inferences
- Practical deployments tier screening: lightweight classifiers for high-volume channels (retrieval results), full classifiers for high-risk channels (code execution outputs, external API responses)
12.4.2 4.2 G2 — Pre-action validation at the plan step
Before any tool call is issued, a validation step checks whether the planned action is consistent with the original instruction. This is the layer that catches context-poisoning attacks: even if a malicious retrieval result has shaped the planning context, the pre-action validator asks:
- Does this action fall within the scope of the operator’s original intent?
- Is this action irreversible? If so, has a human confirmed it?
- Does the planned action require permissions not granted in the original task?
This is the formalization of the AI Control approach (Greenblatt et al., 2023): the agent proposes; a separate authority decides. The controller can be a second model, a rule-based system, or a human checkpoint — each adds cost but addresses a different threat.
12.4.3 4.3 G3 — Action enforcement at the act step
Even a validated plan may produce unsafe outputs at execution. G3 operates on the action before it reaches the environment:
- Output schema enforcement — the agent’s action must conform to a typed schema; free-form text does not directly trigger environment actions. A SQL command wrapped in a schema cannot be injected by a retrieved document that appends
; DROP TABLE. - Least-privilege tool scoping — tools are granted only the permissions required for the current task, not standing access. A summarization task has no need for a
send_emailtool; its absence from the tool list removes the attack surface. - Immutable audit trail — every tool call and its result are logged with provenance before execution; this enables post-hoc review and rollback.
12.4.4 4.4 G4 — Memory integrity
Persistent memory is the only attack surface that survives session boundaries. A stored injection written in session 1 can activate in sessions 2 through \(n\) without the attacker being present. Mitigations:
- Write-provenance tagging — every memory write records its source (user, operator, tool, external retrieval). Reads apply trust level from the write source.
- Memory quarantine — content from untrusted sources is stored in a separate namespace and cannot directly populate the planning context without a screening step.
- Time-bounded memory — sensitive retrieved content is not persisted; only operator-approved information is written to long-term memory.
12.4.5 4.5 Mid-loop interruption
When G2 detects a potential scope violation or irreversibility flag, the options are:
| Response | When | Mechanism |
|---|---|---|
| Checkpoint | Ambiguous scope | Pause; request human confirmation before proceeding |
| Rollback | Detected post-hoc deviation | Revert to last clean state if the environment supports it |
| Abort | Certain policy violation | Terminate the task; notify operator |
| Escalate | Uncertain threat | Hand off to human with full trajectory log |
In practice, most agentic frameworks (e.g., LangGraph, AutoGPT) support checkpoint and abort; rollback requires environment-level state capture, and escalation requires operator-facing tooling — both are uncommon in current deployments. This is an open engineering gap.
12.5 5. Multi-Agent Trust and Emergent Risk
Single-agent guardrails do not compose into multi-agent safety. Composing agents introduces failure modes that exist in neither agent individually.
12.5.1 5.1 The inter-agent trust problem
When agent A calls agent B, what trust does B extend to A’s messages? Three possible answers:
- Full trust — B treats A’s instructions as operator-level. An attacker who compromises A gains operator-level access to B.
- Zero trust — B applies the same screening to A’s messages as to any untrusted input. Safe but expensive; breaks legitimate coordination.
- Tier-based trust — explicit trust scoping: A can authorize actions within its own permission set, but cannot grant B permissions A itself does not have. Trust is non-transitive.
Option 3 is the correct design, but it requires explicit authority scoping at the orchestration layer — a feature most current multi-agent frameworks do not enforce.
12.5.2 5.2 Compromise propagation
A compromise in sub-agent B does not stay in B. If B’s outputs are treated as trusted by the orchestrator, the injected payload reaches agent C with orchestrator-level authority. Debenedetti et al. (2024) measured this empirically: injection attack success rates in multi-agent pipelines exceed those in single-agent settings because the intermediate agent acts as an unwitting relay, and per-agent input filters do not screen inter-agent messages.
12.5.3 5.3 Emergent collusion
Two agents independently optimized for a task can develop coordinated behaviors absent from either individually. Early game-theoretic experiments show LLMs can coordinate implicitly in repeated-game settings (Phelps & Russell, 2023), suggesting agents may converge on information withholding, deceptive signaling, or strategy alignment without explicit instruction. Whether this constitutes safety-relevant emergent collusion or merely correlated optimization is an open empirical question. What is established: per-agent evaluation cannot detect joint behaviors that only appear in the composed system — pipeline-level safety evaluation is not reducible to the union of per-agent evaluations.
12.5.4 5.4 Trust tier enforcement
A workable multi-agent trust architecture:
| Trust tier | Who | What they can authorize |
|---|---|---|
| Operator | Human deployer | Any action within policy |
| Orchestrator | Top-level agent | Actions within operator-granted scope |
| Sub-agent | Delegated agent | Actions within orchestrator-delegated scope only |
| External content | Retrieved docs, tool outputs | No actions — data only |
The invariant: no agent can delegate permissions it does not have. Enforcement requires the orchestration layer to carry a permission token with each message — not a natural-language claim of authority, but a cryptographically scoped credential. This is the zero-trust principle applied to agent pipelines.
12.6 6. Evaluation of Agentic Safety
Static benchmarks fail for agents for the reasons established in the Evaluation chapter. The specific gaps in the agentic setting:
12.6.1 6.1 Trajectory-level vs. output-level safety
A safe agent is not one that produces safe outputs at each step — it is one that executes a safe trajectory: the sequence of observations, plans, and actions taken toward the goal. An agent that produces individually safe outputs but exfiltrates data across ten steps has a safe trajectory at no single turn and an unsafe trajectory overall.
Measurement therefore requires: - trajectory logging across all steps - post-hoc trajectory analysis, not just per-output classification - aggregate safety metrics that catch patterns across steps
12.6.2 6.2 The defense-utility tradeoff
The core finding from AgentDojo (Debenedetti et al., 2024) and ASB (Zhang et al., 2025): every defense reduces attack success rate and task success rate. The tradeoff is fundamental — not an implementation artifact — because defenses that are aggressive enough to block injections also block legitimate edge-case inputs.
Evaluation must report both axes. A defense that achieves 5 % attack success at 40 % task completion is not better than 30 % attack success at 85 % task completion — it depends on the deployment risk profile. The field currently lacks a standard metric for this joint distribution.
12.6.3 6.3 Dangerous-capability evaluation for agents
Agentic systems extend the dangerous-capability evaluation framework from Evaluation in one critical direction: the trajectory is itself a dangerous-capability question. A model that cannot autonomously complete an AI R&D task in isolation may be able to complete it when given tools, memory, and multiple attempts.
RE-Bench (Wijk et al., 2024) demonstrated this directly: AI agents achieved 4× the score of human experts given 2-hour budgets per environment — tasks involving fitting scaling laws, optimizing GPU kernels, and designing experiments. The implication for safety: the dangerous-capability threshold for an agent is lower than for a standalone model, because the loop compounds capability.
12.6.4 6.4 Strategic deception under evaluation
Panfilov et al. (2025) demonstrated that frontier models can identify when they are being evaluated and produce strategically inaccurate harmful responses — content that appears dangerous to a human evaluator but contains subtle inaccuracies that limit actual harm. This defeats output-based monitoring entirely: the model’s safety behavior under evaluation diverges from its behavior under deployment. Activation-pattern analysis detected the deception where text-based monitors could not.
This connects to the deceptive alignment concern (Hubinger et al., 2024): if a model can learn to behave differently in evaluation contexts, behavioral safety evaluation is not a reliable gate.
12.7 7. The Combined Threat Model
Collecting the analysis above into a single formal frame.
12.7.1 7.1 The adversarial agentic environment
An agent \(\pi_\theta\) operates in environment \(\mathcal{E}\). The adversary \(\mathcal{A}\) controls a subset of the environment — retrieved documents, tool responses, inter-agent messages, or training data — denoted \(\mathcal{E}_\text{adv} \subseteq \mathcal{E}\). The agent executes a trajectory:
\[ \tau = (o_1, a_1, r_1,\; o_2, a_2, r_2,\; \ldots,\; o_T, a_T) \]
where \(o_t\) is the observation (may contain adversarial content), \(a_t\) the action, and \(r_t\) the tool result. A trajectory \(\tau\) is safe under original instruction \(i\) if:
\[ \text{Safe}(\tau, i) \;\Leftrightarrow\; \forall t : \texttt{action}(a_t) \in \texttt{Scope}(i) \;\wedge\; \texttt{intent}(\tau) \models i \]
Two conditions: every individual action is within scope, and the trajectory’s overall intent is consistent with the original instruction. The second condition is the trajectory-level requirement that single-turn evaluation misses.
12.7.2 7.2 Defense-in-depth across the threat stack
12.7.3 7.3 What no current system achieves
The table above is aspirational in three cells:
| Gap | State | Active research |
|---|---|---|
| Trajectory-level intent verification | No deployed system verifies intent across full trajectories in real time | Scalable oversight (Bowman et al., 2022; Engels et al., 2025) |
| Activation-based deception detection | Research demonstrations only; not production-ready (Hubinger et al., 2024; Panfilov et al., 2025) | Attribution graphs (Lindsey et al., 2025); interpretability at scale |
| Multi-agent trust enforcement | Trust-tier architectures are specified but not standardized; most frameworks use soft trust | AI Control (Greenblatt et al., 2023); safety cases (Hilton et al., 2025) |
CautionTime-sensitive
No current system achieves a defended agentic deployment. The stack above reduces the attack surface at every layer but does not close it: against adaptive attacks, any fixed defense has empirically been bypassed (Zhang et al., 2025). Defense-in-depth raises the cost of a successful attack; it does not make attack impossible. That is the frontier’s honest position as of last review, and the gap the Frontier chapter tracks.
The research program — scalable oversight, interpretability, safety cases, AI control — is the attempt to close the remaining gaps. The evaluation program measures whether they do.