10 Systemic Safety & Governance
- Systemic safety is risk not contained in any single system: malicious use, structural harm, malfunction at scale.
- Technical safety reduces one deployment’s failure probability. Governance controls the hazard rate and the cross-deployment sum.
- The instruments run soft → hard: frontier frameworks (voluntary) → standards → disclosure → regulation (binding).
- Safety cases make the argument auditable, and expire when their assumptions drift.
The four preceding pillars work at the level of a model or an agent. Systemic safety is the fifth: risk that is not contained in any single system — harm that emerges from deployment at scale, from interaction with society, and from the incentives of the actors who build and run these systems. Its toolkit is not loss functions and monitors but frameworks, standards, and disclosure — the layer that decides whether and how a capable system reaches the world.
10.1 Why a Separate Pillar
A perfectly aligned, robust, well-monitored model can still produce society-scale harm. Standard risk decomposition makes the aggregate exposure explicit:
\[ \mathcal{R} = \sum_i P(\text{hazard}_i) \times P(\text{safeguard fails} \mid \text{hazard}_i) \times \text{Severity}_i \]
Technical safety work primarily reduces \(P(\text{safeguard fails})\) for a single deployment. Systemic risk is the residual: hazard probability, the tail of severity, and the cross-deployment sum \(\sum_i\) — all of which governance, not engineering, controls.
Read the decomposition as a division of labor: engineering owns one term (\(P(\text{safeguard fails})\)), governance owns the rest (hazard rate, severity tail, the sum over deployments). This is why a perfectly aligned, robust, monitored model can still produce society-scale harm, and why the fifth pillar cannot be solved with a loss function.
- Malicious use — a capable, working-as-intended model lowers the cost of CBRN uplift, offensive cyber operations, or mass disinformation. The harm is in who wields the capability, not in a model failure.
- Structural / systemic risk — diffuse effects no single deployment is responsible for: labor displacement, erosion of the information ecosystem, concentration of power, automation of high-stakes decisions (Bengio et al., 2025; Hendrycks et al., 2023).
- Malfunction at scale — a single fault replicated across millions of deployments becomes a societal exposure, echoing the system-safety lineage of complex socio-technical failure (Dobbe, 2022; Leveson, 1995).
These cut across the technical pillars rather than sitting beside them — which is why governance, not engineering, is the operative instrument.
10.2 Frontier Safety Frameworks
The lab-side response: voluntary commitments tying deployment to capability thresholds. Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and DeepMind’s Frontier Safety Framework share a structure — define dangerous-capability levels (CBRN, cyber, autonomy), commit to evaluations that detect when a model crosses one, and gate deployment on safeguards matched to that level. The evaluation chapter covers the measurement side; here the point is governance: thresholds turn an open-ended risk question into a pre-committed if-capability-then-safeguard rule. Their weakness is also structural — voluntary, self-assessed, and only as strong as the evaluations that trigger them.
A frontier framework binds only as far as its own evaluations do. These commitments are voluntary and self-assessed, and they trigger on capability thresholds detected by evals. An eval that fails to detect a crossed threshold silently voids the commitment. The framework is never stronger than the measurement underneath it.
10.3 Standards & Regulation
The regulatory landscape moves faster than the publication cycle. Entries below reflect status as of last review, so check primary sources before citing in policy or compliance contexts.
Where frameworks are voluntary, standards and law are not.
| Instrument | Type | Jurisdiction | Scope | Status (as of last review) |
|---|---|---|---|---|
| NIST AI RMF + GenAI Profile (National Institute of Standards and Technology, 2023) | Framework | US | Lifecycle risk management (govern, map, measure, manage) | Active; GenAI Profile published 2024 |
| ISO/IEC 42001 (International Organization for Standardization, 2023) | Standard | International | AI management system (certifiable, like ISO 27001) | Published 2023; certifications emerging |
| EU AI Act (European Parliament and Council of the European Union, 2024) | Regulation | EU | Risk-tiered; unacceptable uses prohibited; high-risk uses require conformity assessment; GPAI/systemic-risk obligations | In force 2024; phased compliance 2025–27 |
| US — EO 14110 | Executive order | US | Safety evaluation, red-teaming, reporting requirements | Rescinded January 2025 |
| US — NIST/CAISI track | Standards track | US | Successor posture; AI safety standards via CAISI | Active; evolving |
The through-line: standards convert the technical pillars into auditable process and regulation makes some of that process mandatory — the mechanism by which safety research becomes deployment constraint.
10.4 Disclosure & Safety Cases
Governance needs evidence. The disclosure layer supplies it:
Model & system cards — documented capabilities, limitations, evaluations, and intended use; the baseline transparency artifact.
Safety cases (Hilton et al., 2025) — structured, auditable arguments that a specific system is safe enough to deploy in a specific context. The formal structure: given evidence set \(E\) and assumptions \(A\), demonstrate that safety claim \(C\) follows:
\[ (E,\, A) \;\vdash\; C \]
\(E\) assembles the technical results — an interpretability finding, a control evaluation bound, an eval score; \(A\) makes assumptions explicit (deployment context, threat model, capability ceiling); \(C\) is a specific deployment-context safety property. Safety cases fail when \(A\) drifts — capability improvement can invalidate a previously-sound case — making them living artifacts, not one-time arguments.
A safety case expires. \((E, A) \vdash C\) holds only while the assumptions \(A\) hold, and capability improvement invalidates assumptions silently, without any evidence in \(E\) changing. A safety case is a claim with a shelf life rather than a certificate. Treating a past approval as standing authorization is the governance analogue of trusting a stale eval.
10.5 Ecosystem & Compute Governance
Governance also targets the supply chain and the information environment:
- Provenance & content authenticity — watermarking and C2PA-style signed provenance to keep the information ecosystem legible as synthetic content scales (tooled in Tools & Frameworks).
- Compute & hardware-enabled mechanisms — using the concentration of frontier training in identifiable compute as a governance lever (reporting thresholds, hardware-level verification).
10.6 Open Problems
| Problem | Why it persists |
|---|---|
| Voluntary-framework enforceability | Frontier frameworks are self-assessed; commitments bind only as far as the triggering evaluations are valid and honored |
| Pace mismatch | Capability moves faster than standards and law; binding instruments lag the systems they govern |
| Threshold validity | Capability thresholds are only as good as the evals that detect them — and evals degrade as the subject improves |
| Diffuse / structural harm | Systemic risks have no single responsible deployment, so no single actor or control internalizes them (Hendrycks et al., 2023) |
| Provenance robustness | Watermarks and provenance signals are removable or forgeable; the authenticity problem outpaces its tooling |
Systemic safety is where the book’s technical content meets the world: it is the layer that decides whether an aligned, secure, interpretable, monitored system is actually allowed to act — and the layer that must absorb the harms no single system was built to prevent.