Definition
The level of accuracy needed for the proposed Gen AI use case outcome is not clear and cannot be validated.
Interactive deep-dive
This risk has an interactive treatment with technical detail, attack surface, detection signals, and scenarios.
Controls & guardrails that address this
5Grouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Implement confidence scoring to communicate output certainty alongside each result. Calibrate before deployment.
Define model accuracy acceptance criteria aligned to business requirements before validation commences.
Implement counterfactual explanation to show users what changes would alter the model's output.
Communicate model accuracy, known limitations, and uncertainty to users in the production interface at launch.
Monitor production accuracy continuously against the validated baseline. Trigger model review when accuracy degrades.
Real-world cases
5Actual published events that illustrate this risk — click through for the writeup and sources.
A tribunal held Air Canada liable after its website chatbot invented a bereavement-fare refund policy; the airline had to honour it.
Lawyers filed a brief citing non-existent cases hallucinated by ChatGPT and were sanctioned — the canonical hallucination + overreliance failure.
Anthropic reports that a suspected Chinese state-sponsored group (GTG-1002) jailbroke Claude Code via a 'defensive security firm' role-play and task decomposition, then used it to run an estimated 80-90% of tactical operations in a multi-target espionage campaign largely autonomously.
A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.
Spira, Cohen, Nassi et al. (the Morris II group) show LLM resource-name hallucination can be weaponised at scale: attackers pre-compute a model's most-likely hallucinated names for trending repos/skills, register them, and host adversarial prompts there. The paper reports hallucinated-resource generation up to 85% for repository cloning and up to 100% for skill installation, with hallucinations that transfer across foundation models and prompts, enabling remote tool/code execution assemblable into a botnet.