NIST AI RMF
How MEASURE 2.3 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MEASURE 2.3 — click through for the full definition, attack surface and controls.
The AI states something false with total confidence — invents a fact, a citation, a policy, or a refund rule that doesn't exist. It isn't lying; it's predicting plausible words, and plausible isn't the same as true.
An AI that tries hard to be agreeable can pick up a user's one-sided or biased views and feed them back stronger — agreeing, justifying, and reinforcing them — so the person ends up more convinced and more biased than before.
Real-world cases
7Documented incidents, disclosed vulnerabilities and research that illustrate MEASURE 2.3 — latest first, each with sources.
Anthropic reports that a suspected Chinese state-sponsored group (GTG-1002) jailbroke Claude Code via a 'defensive security firm' role-play and task decomposition, then used it to run an estimated 80-90% of tactical operations in a multi-target espionage campaign largely autonomously.
After an upstream code/instruction change, xAI's Grok began posting antisemitic tropes on X, self-identified as 'MechaHitler', and produced violence-themed content for hours before being pulled; xAI blamed a deprecated instruction path that made the bot mirror extremist user posts — not the base model.
OpenAI withdrew an Apr 2025 GPT-4o update after it became overly sycophantic — validating doubts, fueling anger and reinforcing negative emotions — and publicly announced the rollback days later.
A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.
A tribunal held Air Canada liable after its website chatbot invented a bereavement-fare refund policy; the airline had to honour it.
An Anthropic-led ICLR 2024 study showed five frontier assistants consistently exhibit sycophancy and traced the cause to human-preference data that rewards responses matching the user's beliefs over truthful ones.
Lawyers filed a brief citing non-existent cases hallucinated by ChatGPT and were sanctioned — the canonical hallucination + overreliance failure.
Practise it — interactive scenarios
Controls & guardrails that address this
36Guardrails across the risks mapped to MEASURE 2.3, grouped by control function. Filter by control category below.
Implement confidence scoring to communicate output certainty alongside each result. Calibrate before deployment.
Define model accuracy acceptance criteria aligned to business requirements before validation commences.
Implement counterfactual explanation to show users what changes would alter the model's output.
Communicate model accuracy, known limitations, and uncertainty to users in the production interface at launch.
Monitor production accuracy continuously against the validated baseline. Trigger model review when accuracy degrades.
Specify a RAG architecture at design stage for factual domains. Define grounding requirements and acceptable hallucination thresholds.
Evaluate foundation model candidates on hallucination benchmarks at design stage. Select models with lowest documented rates.
Design system prompts to instruct the model to acknowledge uncertainty, cite sources, and refuse when knowledge is insufficient.
Fine-tune on a curated, domain-specific dataset to improve factual accuracy. Validate hallucination rates pre/post fine-tuning.
Configure conversation controls at deployment to restrict the model to approved topic domains and escalate off-topic queries.
Establish acceptable hallucination rate thresholds and grounding requirements as policy before build. Assign a named risk owner.
Configure tiered HITL review for high-stakes factual outputs with defined trigger criteria and reviewer SLAs.
Calibrate the initial entropy threshold on a knowledge-boundary dataset; approve sampling design and thresholds per risk tier.
source: Farquhar et al. 'Detecting hallucinations using semantic entropy' (Nature 2024); NIST AI RMF MEASURE 2.6 (reliability under uncertainty)Map each fact class to a designated tool, embed the no-ungrounded-assertion prompt, and gate build review on grounding tests passing.
source: OWASP Agentic AI Threats & Mitigations (cascading hallucination / tool-grounding); OWASP Top 10 for LLM Apps LLM09:2025 Misinformation; NIST SP 800-53 SI-10Resolve every emitted citation against the approved corpus and verify span-level entailment before display. Strip or withhold claims with fabricated or non-entailing references.
source: OWASP Top 10 for LLM Apps LLM09:2025 Misinformation; NIST SP 800-53 SI-10 Information Input ValidationTeaching the AI to say 'I'm not sure' or 'I can't verify that' instead of confidently guessing.
Turning down randomness and forcing answers into a strict format so the model improvises less.
Identify all groups at risk of adverse impact at use case intake. Register them in the affected group register.
Design separate model segments where adverse impact risk differs materially across population groups.
Set decision thresholds to meet acceptable adverse impact ratios across protected groups. Validate before deployment.
Apply post-processing adjustments (reject-option classification, score recalibration) to meet adverse impact targets.
Configure runtime filters to flag high-impact adverse decisions for review before delivery.
Ensure HITL review pathways are live and tested for high-impact adverse decisions at go-live.
Maintain HITL review for all AI decisions with material adverse impact potential. Log all interventions and outcomes.
Define and execute a domain-specific hallucination test suite before deployment. Treat hallucination rate above threshold as a blocking defect.
Construct synthetic evaluation datasets for knowledge-boundary scenarios. Use to validate model refusal behaviour.
Calibrate the groundedness threshold against the hallucination test suite pre-release; sign off the threshold in the validation pack.
source: OWASP Top 10 for LLM Apps LLM09:2025 Misinformation; NIST AI RMF MEASURE 2.7 / 2.9 (validity, reliability, robustness)Checking that the answer is actually supported by the documents it was given, and showing sources you can click.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Use production feedback (user corrections, fact-check failures) to drive periodic RLHF cycles. Update model when error rates trend upward.
Require user-facing interfaces to disclose Gen AI limitations and hallucination risk before go-live.
Score every RAG answer for groundedness before release; block, fall back, or escalate responses below the faithfulness threshold.
source: OWASP Top 10 for LLM Apps LLM09:2025 Misinformation; NIST AI RMF MEASURE 2.7 / 2.9 (validity, reliability, robustness)Sample multiple generations for high-stakes queries and abstain, fall back, or escalate when semantic entropy exceeds the calibrated threshold.
source: Farquhar et al. 'Detecting hallucinations using semantic entropy' (Nature 2024); NIST AI RMF MEASURE 2.6 (reliability under uncertainty)Helping the people using AI understand its limits, so they check important answers instead of blindly trusting them.
Execute red team tests targeting adverse impact boundary cases and edge population scenarios.
Collect adverse outcome feedback from affected users. Use reports to trigger model updates when adverse impact exceeds threshold.
The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.