AIG-017 AI Model Explainability
Description
For AI systems that produce decisions or recommendations affecting users, documentation of the model's explainability capabilities exists. This includes: the type and level of explanation available (feature attribution, confidence scores, decision paths), known limits on generalizability, and guidance for operators on how to interpret outputs. Where explanations are not technically feasible, this limitation is documented, disclosed to users, and factored into human oversight design.
Rationale
Unexplainable AI outputs prevent operators from identifying errors, challenging decisions, or understanding when to override; explainability is also a legal requirement for certain automated decisions.
Framework Mappings (4)
| EU-AI-Art.13.3 | Transparency — Mandatory Content of Instructions for Use | partial |
| GDPR-Art.22 | Automated Decision-Making and Profiling | partial |
| MAP 2.2 | AI System Knowledge Limits Documentation | full |
| MEASURE 2.9 | AI Model Explainability and Validation | full |
Evidence (1)
Model explainability documentation describing the explanation type and level available (feature attribution, confidence scores, decision paths), known limits, and operator guidance for interpreting outputs.
Example: Model Card — Credit Risk Model v2 (MLflow model card), section 'Explainability': SHAP feature importance explanations available via API, confidence score returned with each prediction, documented precision at confidence thresholds, and operator guidance on interpreting low-confidence outputs
Test: Request explainability documentation for AI systems making decisions affecting users. Verify: (1) explanation type and level are specified, (2) known limits on generalisation are stated, (3) operator guidance for interpreting outputs is included, (4) where explanations are not technically feasible, this limitation is explicitly stated, disclosed in user-facing documentation, and factored into the human oversight design for that system.
Questions (2)
Is the explainability capability of your AI systems documented, including the type of explanation available and known limits?
Unexplainable AI outputs prevent operators from identifying errors or challenging decisions. Documentation should specify what explanations are available (feature attribution, confidence scores, decision paths) and, where explanation is not technically feasible, state this limitation explicitly.
Which of the following explanation types are available for AI systems that make or inform decisions affecting users?
Organisations must select at least one applicable option. If explainability is not feasible, this must be documented, disclosed to users, and factored into human oversight design — selecting only the final option without strong oversight controls is a risk flag.