AIG-026 AI Security and Adversarial Robustness
Description
AI systems are evaluated for AI-specific security vulnerabilities as part of the V&V process and periodically in production. Evaluation covers: data poisoning attacks, model poisoning, adversarial examples, model inversion and extraction attacks, and confidentiality attacks. For GPAI models with systemic risk designation, adversarial testing is conducted per standardised protocols. Security findings are tracked to remediation. AI security evaluation is distinct from and supplementary to standard application security testing.
Rationale
AI-specific attacks (adversarial inputs, poisoning, extraction) are not detected by conventional SAST/DAST tooling; they require dedicated evaluation.
Framework Mappings (4)
| EU-AI-Art.15.3 | Accuracy, Robustness and Cybersecurity — Cybersecurity Against AI-Specific Attacks | full |
| EU-AI-Art.55.1 | Systemic Risk Obligations — Adversarial Testing and Model Evaluation | full |
| MEASURE 2.6 | AI System Safety Risk Evaluation | partial |
| MEASURE 2.7 | AI System Security and Resilience Evaluation | full |
Evidence (2)
AI-specific security evaluation report covering data poisoning, adversarial examples, model inversion, and extraction attack vectors, produced before deployment and periodically in production.
Example: AI Security Assessment — Recommendation Engine v2 (Confluence, dated 2025-10-12): adversarial robustness testing results (FGSM, PGD attacks), model inversion test result, data poisoning resilience test, findings tracked in security backlog AI-SEC-2025-003 through AI-SEC-2025-007
Test: Request AI security evaluation reports for a sample of Tier 2+ AI systems. Verify: (1) evaluation covers AI-specific attack classes (poisoning, adversarial examples, inversion/extraction) distinct from SAST/DAST results, (2) findings are tracked to remediation with severity and status, (3) report is dated before deployment or within the last 12 months, (4) for GPAI systemic risk models, adversarial testing was conducted per standardised protocol with a named methodology.
Third-party red team assessment or AI security audit report for high-risk AI systems, providing independent validation of AI-specific security posture.
Example: Red Team Assessment Report — GPAI Foundation Model (SecurityVendor Ltd, 2025-09-15), covering adversarial robustness, jailbreak resilience, model extraction, membership inference, and training data leakage, with CVSS-equivalent AI risk ratings and remediation recommendations
Test: Request the most recent third-party AI security assessment. Verify: (1) assessment scope covers AI-specific attack vectors beyond conventional application security, (2) assessor is independent of the internal team, (3) findings are formally tracked and remediation is evidenced, (4) report date is within the last 24 months for high-risk systems, (5) for systemic risk GPAI models, confirm adversarial testing was performed per the EU AI Act Art. 55.1 obligation.
Questions (2)
Are AI systems evaluated for AI-specific security vulnerabilities — such as data poisoning, adversarial examples, and model extraction — as part of your security testing programme?
AI-specific attacks are not detected by conventional SAST/DAST tooling. Evaluation must be explicitly scoped to AI attack classes and conducted separately from standard application security testing.
Which AI-specific attack classes are evaluated in your AI security testing programme?
Coverage of all six attack classes characterises a mature AI security evaluation programme. For GPAI models with systemic risk designation, adversarial testing per standardised protocols is an EU AI Act Art. 55.1 obligation.