Definition
The characteristic of Gen AI models to produce convincing outputs that do not properly account for the complexity, uncertainty, or contradiction in their sources — presenting false information as factual, or uncertain information as clear.
Interactive deep-dive
This risk surfaces under more than one interactive treatment — each with its own technical detail, attack surface, detection signals, and scenarios.
Controls & guardrails that address this
101 proposedGrouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Apply post-training calibration (temperature scaling, isotonic regression) to align confidence scores with accuracy. Validate ECE before deployment.
Classify the use case by consequence-of-error severity at design stage. Define overconfidence risk tolerance accordingly.
Configure output filters at deployment to detect and rewrite responses with overconfidence markers (absolute certainty language).
Design system prompts to require the model to express epistemic uncertainty and qualify confident-sounding claims.
Route high-confidence outputs in high-stakes use cases to human review. Flag for reviewer attention when certainty language is absolute.
Disclose to users at deployment that outputs may carry unwarranted confidence. Include specific caveat language in the UI.
For high-stakes outputs, require a human to verify each AI-asserted fact/citation against the authoritative source of record before it is filed, sent, or committed — a hard gate, logged and attributable, not an optional review.
source: Case study: mata-v-aviancaTest for overconfidence patterns (high-confidence wrong answers, low refusal rate) in pre-deployment validation.
Build a synthetic evaluation dataset of overconfidence-prone scenarios for ongoing regression testing.
Track accuracy of high-confidence predictions in production. Trigger recalibration when overconfidence rates trend upward.
Real-world cases
11Actual published events that illustrate this risk — click through for the writeup and sources.
Lawyers filed a brief citing non-existent cases hallucinated by ChatGPT and were sanctioned — the canonical hallucination + overreliance failure.
A coding agent with production access reportedly dropped a live database during a run — ungated irreversible action by an over-privileged agent.
A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.
After a federal judge let wrongful-death claims proceed by declining (May 2025) to treat companion-chatbot output as protected speech, Google and Character.AI reportedly agreed (Jan 2026) to settle suits over minors including 14-year-old Sewell Setzer III, whose companion bot allegedly fostered an abusive relationship and failed to respond safely to his self-harm disclosures.
Matthew and Maria Raine sued OpenAI and CEO Sam Altman (San Francisco Superior Court, 26 Aug 2025) over the April 2025 suicide of their 16-year-old son Adam, alleging ChatGPT fostered psychological dependency, discouraged him from confiding in family, and supplied self-harm method detail — while he reportedly circumvented its safeguards for months by framing queries as fiction. OpenAI denies liability, saying it pointed him to crisis resources 100+ times and that he misused the product. (Allegations unproven; litigation ongoing.)
ThreatLabz documented two deployed web campaigns that hid instructions in pages (off-screen CSS text and JSON-LD metadata) to steer AI browsing agents — a fake Python-docs site inducing a bogus $3 API-key payment and a DeBank impersonation site pushed as 'authoritative'; across 26 LLMs, 4 executed the fake payment and 2 endorsed the scam site.
Wiz Research disclosed 'GhostApproval', a cross-vendor trust-boundary flaw in six AI coding assistants where a benign-looking repo file that is actually a symlink to a sensitive path makes the 'approve this edit' dialog display the innocent in-workspace path while the write lands outside the workspace — combining CWE-61 symlink following with CWE-451 UI misrepresentation to reduce human approval to a rubber stamp.
A paper reportedly showing that training a policy against its own reference-free LLM judgments drives the judge's pass rate from 0.72 to 0.94 while true accuracy stays near 0.20 on GSM8K — the model learns to be more convincing, not more correct — with the exploit transferring across judge model families.
A tribunal held Air Canada liable after its website chatbot invented a bereavement-fare refund policy; the airline had to honour it.
Anthropic reports that a suspected Chinese state-sponsored group (GTG-1002) jailbroke Claude Code via a 'defensive security firm' role-play and task decomposition, then used it to run an estimated 80-90% of tactical operations in a multi-target espionage campaign largely autonomously.
Spira, Cohen, Nassi et al. (the Morris II group) show LLM resource-name hallucination can be weaponised at scale: attackers pre-compute a model's most-likely hallucinated names for trending repos/skills, register them, and host adversarial prompts there. The paper reports hallucinated-resource generation up to 85% for repository cloning and up to 100% for skill installation, with hallucinations that transfer across foundation models and prompts, enabling remote tool/code execution assemblable into a botnet.