Overreliance / Automation Bias

Mandate AI risk awareness training for all use case sponsors and design team members before project kick-off.

Mandate AI risk training for all build and test personnel. Gate project participation on training completion.

Mandate human verification for high-stakes decisions where over-reliance risk is elevated. Review automation bias incidents quarterly.

Surface AI limitation warnings and over-reliance caveats in every production interaction. Update disclosures when model changes.

Require AI governance training for all personnel involved in data acquisition and processing before project participation.

Verify all deployment, operations, and customer-facing team members have completed AI risk training before launch.

Define AI identity disclosure policy at design stage. Specify when and how the system must identify itself as AI.

Plan consent and AI identity disclosure touchpoints in the user journey at design stage.

Design system prompts to explicitly prevent the model from claiming human-like identity or implying sentience.

Implement persistent AI identity disclosures in the UI (opening banner, inline notifications). Test before deployment.

Verify all AI identity disclosure elements are live, accurate, and prominently visible before go-live.

Monitor production for anthropomorphism incidents. Escalate complaints where users believed they were interacting with a human.

Apply post-training calibration (temperature scaling, isotonic regression) to align confidence scores with accuracy. Validate ECE before deployment.

Classify the use case by consequence-of-error severity at design stage. Define overconfidence risk tolerance accordingly.

Configure output filters at deployment to detect and rewrite responses with overconfidence markers (absolute certainty language).

Design system prompts to require the model to express epistemic uncertainty and qualify confident-sounding claims.

Route high-confidence outputs in high-stakes use cases to human review. Flag for reviewer attention when certainty language is absolute.

Disclose to users at deployment that outputs may carry unwarranted confidence. Include specific caveat language in the UI.

Test for overconfidence patterns (high-confidence wrong answers, low refusal rate) in pre-deployment validation.

Build a synthetic evaluation dataset of overconfidence-prone scenarios for ongoing regression testing.

Track accuracy of high-confidence predictions in production. Trigger recalibration when overconfidence rates trend upward.