13  Tools & Frameworks

14 Tools & Frameworks

Stub. The applied layer — what practitioners actually deploy. To catalog (with what each does, where it sits in the attack surface, and maturity):

• Guardrails / runtime — NVIDIA NeMo Guardrails, Llama Guard (Inan et al., 2023), ShieldGemma, Guardrails AI, Constitutional Classifiers (Sharma et al., 2025), interface firewalls (Huang et al., 2025).
• Evaluation harnesses — Inspect (UK AISI), AgentDojo (Debenedetti et al., 2024), Agent Security Bench (Zhang et al., 2024), HELM, lm-evaluation-harness.
• Red-teaming / attack — garak, PyRIT, Giskard.
• Interpretability — TransformerLens, Gemma Scope SAEs (Lieberum et al., 2024).
• Provenance / content authenticity — C2PA, watermarking toolkits.

Selection criteria, integration patterns, and a maturity rating per tool to follow.

Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate attacks and defenses for LLM agents. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks.

Huang et al. (2025). Indirect prompt injections: Are firewalls all you need, or stronger benchmarks? Advances in Neural Information Processing Systems (NeurIPS).

Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., & Khabsa, M. (2023). Llama guard: LLM-based input-output safeguard for human-AI conversations. arXiv Preprint arXiv:2312.06674.

Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., & Nanda, N. (2024). Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv Preprint arXiv:2408.05147.

Sharma, M. et al. (2025). Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv Preprint arXiv:2501.18837.

Zhang, H. et al. (2024). Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. arXiv Preprint arXiv:2410.02644.