16  Tools & Frameworks

Living document. This catalog covers tools prominent at the time of last review and is not exhaustive. The tooling landscape evolves rapidly — products are acquired, deprecated, or superseded, and new categories emerge as the attack surface grows. Entries will be updated each quarterly review. Maturity ratings reflect deployment track record, not capability claims.

What practitioners actually deploy. This catalog maps each tool to where it sits in the attack surface and rates it by maturity. The organizing question is placement: a guardrail at the input channel and an audit log at the action layer defend different things, and a stack is only as strong as its thinnest layer.

16.1 Where Tools Sit

Tool categories map onto distinct points of the agent pipeline; defense-in-depth means covering every point, not hardening one.

16.2 Guardrails & Runtime

Inline controls screening inputs, outputs, and actions at deployment — the L1/L5 layers of the defensive stack.

Tool	Does	Maturity
Llama Guard (Inan et al., 2023)	LLM classifier scoring prompts/responses against a configurable harm taxonomy	Production
NeMo Guardrails	Programmable rails (topical, safety, execution) around an LLM app via a dialog/flow spec	Production
ShieldGemma / Guardrails AI	Open content-safety classifiers and a validation framework for structured/safe output	Production
Constitutional Classifiers (Sharma et al., 2025)	Classifiers trained on adversarial jailbreak variants against natural-language policies	Emerging
Interface firewalls (Bhagwatkar et al., 2025)	Dual-LLM / policy-model isolation so the primary model never sees raw untrusted content	Research → early adoption

16.3 Evaluation Harnesses

Infrastructure for measuring safety and capability — the runnable side of Evaluation.

Tool	Does	Maturity
Inspect (UK AISI)	Framework for building and running model evaluations, incl. agentic and dangerous-capability evals	Production (institutional)
AgentDojo (Debenedetti et al., 2024)	Dynamic benchmark for prompt-injection attacks/defenses on tool-using agents	Research standard
Agent Security Bench (Zhang et al., 2025)	Formalized attack/defense evaluation across agent operation stages	Research standard
HELM (Liang et al., 2023)	Multi-metric holistic evaluation across scenarios (accuracy, robustness, fairness, …)	Established
lm-evaluation-harness	Standardized capability benchmarking across many tasks/models	De facto standard

16.4 Red-Teaming & Attack

Offensive tooling for probing systems before adversaries do — standing oversight rather than one-time testing.

Tool	Does	Maturity
garak	Vulnerability scanner probing LLMs for jailbreaks, leakage, injection, toxicity	Production
PyRIT	Risk-identification framework for automating red-team attack generation against GenAI	Production
Giskard	Testing/scanning for ML and LLM apps (bias, robustness, injection)	Production

16.5 Interpretability

The instrumentation behind Interpretability — turning mechanistic analysis into shared infrastructure.

Tool	Does	Maturity
TransformerLens	Library for mechanistic-interpretability experiments (activation caching, patching, hooks)	Research standard
Gemma Scope (Lieberum et al., 2024)	Open suite of sparse autoencoders across every layer of an open model	Research standard

16.6 Provenance & Content Authenticity

The ecosystem-governance tooling from Systemic Safety — keeping the information environment legible as synthetic content scales.

Tool	Does	Maturity
C2PA	Open standard for cryptographically signed content provenance/edit history	Emerging (industry adoption)
Watermarking toolkits	Embed detectable statistical signals in model outputs for attribution	Research → early deployment

16.7 Selection Notes

• Match the tool to the layer, not the brand. A guardrail model and a sandbox defend different threats; deploying two tools at the same layer leaves others bare.
• Maturity is a deployment gate. Production tools have stable interfaces and operational track records; research tools are validated in papers, not at scale.
• Evaluation tooling is itself a control. Harnesses and red-team frameworks are how the other layers get verified; an unevaluated guardrail is an assumption, not a defense.

Bhagwatkar, R., Kasa, K., Puri, A., Huang, G., Rish, I., Taylor, G. W., Dvijotham, K. D., & Lacoste, A. (2025). Indirect prompt injections: Are firewalls all you need, or stronger benchmarks? arXiv Preprint arXiv:2510.05244.

Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate attacks and defenses for LLM agents. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks.

Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., & Khabsa, M. (2023). Llama guard: LLM-based input-output safeguard for human-AI conversations. arXiv Preprint arXiv:2312.06674.

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.

Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., & Nanda, N. (2024). Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv Preprint arXiv:2408.05147.

Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., Askell, A., Bailey, N., Benton, J., Bluemke, E., Bowman, S. R., Christiansen, E., Cunningham, H., Dau, A., Gopal, A., … Perez, E. (2025). Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv Preprint arXiv:2501.18837.

Zhang, H., Huang, J., Mei, K., Yao, Y., Wang, Z., Zhan, C., Wang, H., & Zhang, Y. (2025). Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. International Conference on Learning Representations (ICLR).