AIG-028 Hallucination and Factual Accuracy Controls
Description
LLM-based and generative AI systems that produce statements users may rely upon as factual have controls to detect and limit hallucinated or unverifiable outputs. Controls in place include at minimum two of: retrieval-augmented generation (RAG) with source citation, factual grounding evaluation during testing, output disclaimer labelling for unverified content, output confidence scoring, post-processing fact-checking for high-risk use cases. For use cases where factual accuracy is safety-critical, an additional human verification step is required. The organisation's acceptable hallucination rate for each use case is documented.
Rationale
Hallucination is an intrinsic property of large language models not present in classical software; it requires novel and specific controls that no existing framework addresses at an operational level.
Framework Mappings (3)
| EU-AI-Art.15.1 | Accuracy, Robustness and Cybersecurity — Performance Standards | informative |
| MAP 2.2 | AI System Knowledge Limits Documentation | informative |
| MEASURE 2.5 | AI System Validity and Reliability | informative |
Evidence (2)
Configuration evidence showing that at least two hallucination controls are active for LLM-based systems (e.g. RAG source citation config, output disclaimer settings, post-processing fact-checker integration, confidence scoring), with the acceptable hallucination rate documented.
Example: LLM service configuration (AWS Bedrock Knowledge Bases + LangChain chain config): RAG enabled with source citation required, disclaimer appended to all responses not backed by retrieved sources (unverified_content_label: true), max_unverified_claims_per_response: 0 for medical use case, acceptable_hallucination_rate: <1% per monthly eval
Test: Request the LLM system configuration for each generative AI use case. Verify: (1) at least two controls from the defined list are active (RAG+citation, factual grounding eval, disclaimer labelling, confidence scoring, post-processing fact-check), (2) for safety-critical use cases, a human verification step is configured, (3) acceptable hallucination rate is documented per use case, (4) configurations are version-controlled with a review date.
Hallucination evaluation report from testing showing measured hallucination rate for the LLM system against the documented acceptable threshold, using a defined evaluation dataset or benchmark.
Example: Hallucination Evaluation Report — Legal Research Assistant v2 (Confluence, 2026-02-01): evaluated on 500-question benchmark, hallucination rate 0.4% (threshold: <1%), RAG citation accuracy 97.8%, factual grounding score 0.94, result: PASS; evaluation dataset: internal legal QA benchmark v3
Test: Request hallucination evaluation reports for LLM systems producing factual outputs. Verify: (1) evaluation methodology and dataset are documented, (2) measured hallucination rate is reported and compared to the documented acceptable threshold, (3) evaluation was conducted before deployment and at the defined production evaluation interval, (4) for safety-critical use cases, human verification rate is also reported, (5) threshold failures result in a documented remediation decision.
Questions (2)
Do your LLM-based or generative AI systems that produce statements users may rely on as factual have controls in place to detect and limit hallucinated or unverifiable outputs?
Net-new control: hallucination is an intrinsic property of large language models not present in classical software. It requires specific operational controls — existing frameworks do not address this at an actionable level. For safety-critical use cases, additional human verification is required regardless of other controls.
Which hallucination and factual accuracy controls are active for your LLM-based systems?
The control requires at least two active controls from this list. For safety-critical use cases (healthcare, legal, financial decisions), the human verification step is mandatory regardless of what other controls are in place.