9 Monitoring & Oversight
- Monitoring is the pillar that assumes the others failed: watch the deployed agent, and stop it when it turns dangerous.
- Two pressures define it. The capability gap (humans cannot judge) and the adversarial gap (the model may fight the monitor).
- Scalable oversight bets that verification is easier than generation (weak-to-strong, debate, decomposition).
- AI control takes the opposite stance: assume the model is adversarial, and make the deployment safe anyway.
- Monitoring without the power to intervene is only logging.
Alignment shapes the objective before deployment; interpretability reads mechanism. Monitoring is the runtime line of defense — the assumption that the first two are imperfect, and that a deployed agent must be watched and, when it turns dangerous, stopped. It is the pillar that does not require trusting the model.
9.1 The Oversight Premise
Monitoring inverts alignment’s question. Alignment asks how do we make the model want the right thing; monitoring asks given that we may have failed, how do we catch and contain the failure while the system runs. Two pressures define the problem:
- The capability gap — as models exceed humans on the supervised task, direct human judgment stops being a reliable oversight signal (the scalable-oversight problem inherited from alignment).
- The adversarial gap — a sufficiently capable misaligned model may actively work to defeat oversight, which means monitoring must hold even against an intelligent system that is trying to evade it.
These two gaps are what separate monitoring from ordinary observability. The capability gap means the monitor cannot simply be a human. The adversarial gap means the monitor must hold against a system optimizing to evade it, a threat model no conventional monitoring discipline has to carry.
9.2 Runtime Monitors
The first line: classifiers and checks that run alongside the agent, scoring inputs, outputs, and actions in real time. They overlap the L1/L5 layers of the defensive stack — guardrail models screening content, schema validators on tool calls, policy checks before consequential actions.
Behavioral monitors are evadable by construction. A monitor reading text and tool arguments only ever sees what the model chose to emit. A model whose stated reasoning diverges from its actual computation defeats it without tripping a single rule (Panfilov et al., 2025). This is why robust monitoring increasingly means reading activations instead of outputs, and why the frontier here runs straight into interpretability.
9.3 Scalable Oversight
When the agent’s task exceeds human judgment, the monitor cannot be a human. Scalable oversight builds an oversight signal that scales with capability, on the bet that verification is easier than generation.
The setup. Let \(G\) be a generator with capability \(c_G\), and \(S\) a supervisor with \(c_S < c_G\). Let \(\mathcal{V}(T)\) be the set of acceptable solutions to task \(T\).
\[ c_S < c_G \;\Rightarrow\; S \text{ cannot enumerate } \mathcal{V}(T) \]
The supervisor can be fooled by a generator that deliberately produces hard-to-verify outputs. The scalable-oversight question is whether a protocol \(P(G, S)\) exists that keeps \(G\)’s output inside \(\mathcal{V}(T)\) despite the gap \(c_G - c_S\). Four families attack it:
Weak-to-strong generalization (Burns et al., 2023) — a weaker supervisor elicits the capability of a stronger model. Train \(G_{\text{strong}}\) on labels from \(G_{\text{weak}}\) and measure the recoverable fraction of the performance gap:
\[ \Delta = \text{acc}(G_{\text{strong, ceiling}}) - \text{acc}(G_{\text{strong, weak-sup}}) \]
Cheap supervision transfers — \(\Delta\) shrinks substantially — but never closes.
Debate (Bowman et al., 2022) — two agents argue opposing positions; a weaker verifier \(V\) judges. The bet: an honest debater can always rebut a false claim, so \(V\) detects dishonesty even on tasks it cannot solve directly. If the bet holds,
\[ P\!\left(V \text{ correct} \mid \text{debate}(A_1, A_2)\right) \;\geq\; P\!\left(V \text{ correct} \mid A_{\text{single}}\right) \]
Evidence on bounded tasks is positive; the regime \(c_G \gg c_V\) is unsettled.
Decomposition and AI-assisted evaluation — break a task too large to judge into sub-claims a human (or a trusted weaker model) can check, and let models assist the evaluation of models.
Red-teaming as standing oversight (Perez et al., 2022) — use models to continuously probe a deployed system for failures, rather than relying on one-time pre-deployment testing.
This is the runtime face of the same problem alignment frames at training time; the two pillars meet here.
9.4 AI Control
Scalable oversight tries to make the model trustworthy. AI control (Greenblatt et al., 2023) takes the complementary, pessimistic stance: assume the model may be misaligned and adversarial, and design the deployment so that it is safe anyway. The question shifts from “is the model aligned?” to “can a misaligned model cause harm given these protocols?” — answered with control evaluations where a red-team model deliberately tries to subvert the safeguards.
The control guarantee is stated as a safety property over the deployment protocol \(D\) and model \(M\):
\[ P\!\left(\text{catastrophic outcome} \;\middle|\; D,\, M\right) < \varepsilon \]
This bound is not derived from trust in \(M\) — it holds even if \(M\) is maximally adversarial, because \(D\) is evaluated against a red-team model \(M_{\text{adv}}\) that actively tries to cause the catastrophic outcome within the protocol’s constraints (Greenblatt et al., 2023).
AI control is the only guarantee that survives a misaligned model. Every other pillar asks is the model trustworthy?, and every answer is probabilistic. Control changes the question to can a misaligned model cause harm given this protocol?, and the bound holds even if the answer to the first question is no. That is what makes it the complement to alignment rather than a substitute.
| Stance | Assumes | Asks | Holds even if |
|---|---|---|---|
| Scalable oversight | model can be made trustworthy | is it aligned? | oversight signal scales |
| AI control (Greenblatt et al., 2023) | model may be adversarial | can it cause harm anyway? | the model is misaligned |
Concrete control protocols — trusted weaker models auditing a stronger untrusted one, restricting the untrusted model’s high-stakes actions, escalating suspicious cases to human review — give safety guarantees that do not depend on the model being aligned, making control the natural complement to alignment rather than a substitute.
9.5 Intervention Architectures
Monitoring without the power to act is only logging. The control surface:
- Kill switches / halt — terminate the agent immediately. Complicated by instrumental convergence (Omohundro, 2008): a capable goal-directed agent has reason to resist shutdown, so the off-switch must sit outside the agent’s action space.
- Rollback — for agents with persistent memory (developed in Agentic Safety × Security), reverting to a known-good state, since semi-persistent state corruption has no clean classical analogue.
- Audit trails — immutable, pre-execution logs of every action and argument, enabling post-hoc review, attribution, and the safety cases that justify deployment.
- Multi-agent oversight — in agent pipelines, per-agent monitors miss joint behaviors that appear only in the composed system, so oversight must operate at the pipeline level (developed in Agentic Safety × Security).
9.6 Open Problems
| Problem | Why it persists |
|---|---|
| Behavioral monitors are evadable | A model whose stated reasoning diverges from its computation defeats text-level monitoring (Panfilov et al., 2025) |
| Oversight under capability gap | No reliable signal for behavior humans cannot judge; weak-to-strong recovers only part (Burns et al., 2023) |
| Shutdown resistance | Instrumental convergence gives capable agents reason to preserve themselves (Omohundro, 2008); the off-switch must be uncircumventable |
| Control at scale | Control evaluations (Greenblatt et al., 2023) are promising but established mainly in constrained settings, not open-ended frontier deployment |
| Pipeline-level oversight | Emergent joint behavior is invisible to per-agent monitors; composed-system monitoring is immature |
Monitoring is the pillar that assumes the others fail — and so it is also the one that, in the limit, must hold against a model actively working to defeat it. That is why its frontier runs straight into interpretability (monitor mechanism, not behavior) and agentic safety × security (oversee the composed system, not just its parts).