AI Security: Threats, Controls, and Evidence
TL;DR
AI security addresses threats like prompt injection, data exfiltration, model misuse, supply-chain risks, and unsafe tool calls. Controls include input handling, retrieval hardening, capability gating, authZ, output validation, monitoring, incident response, and secure change management. Governance (ISO 42001) ensures these are designed, operated, and reviewed.
Key Facts
AI systems introduce unique threats (instruction override, model misuse).
Layered controls reduce exploitability.
Logging and monitoring are essential for detection and forensics.
Secure change control prevents silent regressions.
Incident handling requires AI-specific playbooks.
Implementation Steps
Threat model → threat model doc.
Design controls → control list, allow-lists.
Test & validate → attack corpus, test logs.
Monitor & alert → log schema, alerts.
IR & lessons learned → IR reports, CAPA.
Glossary
- Model misuse
- Using AI systems for purposes beyond their intended design or capability
- Capability gating
- Controls that limit what actions an AI system can perform
- Output validation
- Verification that AI system outputs meet safety and security requirements
- Retrieval hardening
- Security measures applied to data retrieval processes in AI systems
- Allow-list
- Predefined list of permitted inputs, outputs, or actions
- Telemetry
- Automated collection and transmission of data for monitoring purposes
References
- [1] ISO 42001 AI Management Systems Standard https://www.iso.org/standard/78380.html
- [2] NIST AI Risk Management Framework https://www.nist.gov/itl/ai-risk-management-framework
Machine-readable Facts
[
{
"id": "f-threats",
"claim": "AI systems face threats such as prompt injection, misuse, and unsafe tool calls.",
"source": "https://www.nist.gov/itl/ai-risk-management-framework"
},
{
"id": "f-layers",
"claim": "Layered controls across input, retrieval, tools, and outputs reduce risk.",
"source": "https://www.nist.gov/itl/ai-risk-management-framework"
},
{
"id": "f-logging",
"claim": "Logging and monitoring are essential for detection and forensics in AI security.",
"source": "https://www.iso.org/standard/78380.html"
}
]