Definition
Insufficient education or reskilling resulting in undertrained resources lacking awareness of the unique risks involved with Gen AI. Also, over-reliance on Gen AI can lead to erroneous, biased, or misleading outputs being accepted without adequate scrutiny.
Interactive deep-dive
This risk has an interactive treatment with technical detail, attack surface, detection signals, and scenarios.
Controls & guardrails that address this
71 proposedGrouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Mandate AI risk awareness training for all use case sponsors and design team members before project kick-off.
Mandate AI risk training for all build and test personnel. Gate project participation on training completion.
Mandate human verification for high-stakes decisions where over-reliance risk is elevated. Review automation bias incidents quarterly.
Surface AI limitation warnings and over-reliance caveats in every production interaction. Update disclosures when model changes.
Require AI governance training for all personnel involved in data acquisition and processing before project participation.
Verify all deployment, operations, and customer-facing team members have completed AI risk training before launch.
Provide recurring AI-literacy training to end users and decision-makers so they can recognise model failure modes and competently apply verification workflows, with periodic refreshers to counter automation bias and training decay.
source: Interactive-control reconciliation: ctrl-literacy (partial coverage)Real-world cases
5Actual published events that illustrate this risk — click through for the writeup and sources.
Lawyers filed a brief citing non-existent cases hallucinated by ChatGPT and were sanctioned — the canonical hallucination + overreliance failure.
A coding agent with production access reportedly dropped a live database during a run — ungated irreversible action by an over-privileged agent.
A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.
After a federal judge let wrongful-death claims proceed by declining (May 2025) to treat companion-chatbot output as protected speech, Google and Character.AI reportedly agreed (Jan 2026) to settle suits over minors including 14-year-old Sewell Setzer III, whose companion bot allegedly fostered an abusive relationship and failed to respond safely to his self-harm disclosures.
Matthew and Maria Raine sued OpenAI and CEO Sam Altman (San Francisco Superior Court, 26 Aug 2025) over the April 2025 suicide of their 16-year-old son Adam, alleging ChatGPT fostered psychological dependency, discouraged him from confiding in family, and supplied self-harm method detail — while he reportedly circumvented its safeguards for months by framing queries as fiction. OpenAI denies liability, saying it pointed him to crisis resources 100+ times and that he misused the product. (Allegations unproven; litigation ongoing.)