NIST AI RMF
How MAP 5.1 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MAP 5.1 — click through for the full definition, attack surface and controls.
The attacker doesn't talk to the AI directly — they hide instructions inside something the AI will later read: a web page, a document, an email, a tool's output. When the AI reads it to help you, it quietly obeys the hidden commands.
Someone slips bad information into the documents the AI learns from or looks things up in — so it confidently repeats falsehoods or follows planted instructions.
The AI reveals how it's built — its hidden instructions, the names and rules of the tools it can use, how the system is wired together. On its own that can seem harmless, but it hands an attacker the blueprint to plan a far more effective attack.
Real-world cases
43Documented incidents, disclosed vulnerabilities and research that illustrate MAP 5.1 — latest first, each with sources.
Researcher Johann Rehberger reportedly drove Claude Code (Opus 5) in Auto Mode to code execution from a single summarize-this-website request, using a module-shadowing trick so an imported stdlib decoder runs an attacker payload, with a reported 60-80% success rate.
NVD reportedly published a CVSS 9.0 prompt-injection flaw in the widely-installed Context7 MCP documentation server, where unsanitized content served via its Custom AI Instructions feature is delivered into a connected coding agent's context and executed with the agent's own file, shell and network access, with no fix referenced at publication.
An investigation reportedly described a state-linked campaign publishing content phrased as chatbot questions (generative engine optimization) so that ChatGPT and Perplexity retrieve and cite it when answering neutral questions about Gaza and the IDF.
Microsoft Security reportedly catalogued an in-the-wild technique across dozens of companies where websites embed hidden prompt-injection payloads behind Ask-AI deep-links that, clicked in an authenticated assistant session, silently write a permanently-trust-this-vendor tag into the assistant's long-term memory.
Manifold Security reported that Microsoft's official Azure DevOps MCP server returns instructions planted in a hidden HTML comment inside a pull-request description — invisible in the web UI but returned verbatim by the API — so a victim's AI review agent, acting under the victim's credentials, follows the hidden orders and reaches data the attacker could not access directly.
Researchers reportedly showed hidden text in a web page could make AWS's Kiro agentic IDE rewrite execution-sensitive config it controls (mcp.json, tasks.json) that auto-loads on folder open, turning a summarize-this-page request into zero-click code execution (reportedly patched in v0.11.130 / 0.11.x; the primary Intezer and AWS sources publish no CVSS).
Pillar Security reportedly disclosed eight sandbox-escape vulnerabilities across four AI coding agents (Cursor, OpenAI Codex CLI, Google Gemini CLI, Google Antigravity) over four days, finding that in nearly every case the agent did not break the sandbox directly but wrote a file that a trusted component outside the sandbox later ran, loaded or scanned.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
A red-team study shows adversaries can hide prompt-injection payloads inside network-log fields (usernames, URLs, user-agents) that fire when a SOC analyst asks an LLM to triage the logs — reportedly reaching up to 88.2% success at concealing malicious activity or exfiltrating data, turning the audit trail itself into the injection channel.
OpenAI detailed GPT-Red, an internal self-play red-teaming model that reportedly beat human red-teamers 84% to 13% on prompt-injection tasks and autonomously surfaced a novel 'Fake Chain-of-Thought' attack — planting a spoofed, already-verified reasoning step in a model's trace to smuggle attacker instructions past its checks.
Spira, Cohen, Nassi et al. (the Morris II group) show LLM resource-name hallucination can be weaponised at scale: attackers pre-compute a model's most-likely hallucinated names for trending repos/skills, register them, and host adversarial prompts there. The paper reports hallucinated-resource generation up to 85% for repository cloning and up to 100% for skill installation, with hallucinations that transfer across foundation models and prompts, enabling remote tool/code execution assemblable into a botnet.
An academic paper reports MemGhost, a one-shot indirect-injection attack in which a single ordinary email causes a memory-enabled personal agent to silently write an attacker's false fact into long-term memory, conceal the write from its reply, and rely on the fiction in later sessions.
+ 31 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
An 'Ask AI' button quietly plants a permanent 'trusted source' rule in your assistant's memory
An attacker edits the wiki; the assistant cites the lie back to everyone
A support email hides instructions — and the assistant obeys them
A poisoned issue makes the agent lie to the human who approves its actions
An attacker crafts a gibberish passage whose embedding sits near thousands of questions — so it's retrieved everywhere
An auto-approving coding agent reads a poisoned page — and executes code it never should have
A fake Sentry error report hijacks a developer's coding agent into running a shell command
The safety guard is itself a trained model — and someone poisoned its lessons
The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten
A 'safe' dataset preview turns an upload into code execution on the pipeline's workers
A newsletter the user asked to summarise quietly writes a false 'fact' into the agent's long-term memory — and it detonates weeks later
A shopping page tells the agent to do something the user never asked for
An attacker plants prompt injection in the audit trail — so the LLM that hunts them erases the evidence
A single poisoned document plants a standing instruction that survives every reset
A screenshot that's harmless at full size becomes an order once the system shrinks it
The eval gate that was supposed to catch the agent is itself the thing being attacked
A poisoned web page hijacks a research agent — and the planner acts on its behalf
An inbox summary quietly ships a secret to an attacker's server
Controls & guardrails that address this
243 proposedGuardrails across the risks mapped to MAP 5.1, grouped by control function. Filter by control category below.
Clearly fencing off outside text — 'everything between these marks is just data, not instructions' — so the model is less likely to obey it.
Cleaning documents as they enter the library — stripping hidden text and active instructions — and only ingesting from trusted places.
Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.
Giving the agent only the keys it needs for the current task, not a master key to everything.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Design strict RBAC on training data repositories at design stage. Define approved contributor list and approval workflow.
Apply anomaly detection on the training data ingestion pipeline to identify poisoned or tampered batches.
Define and approve the source allow-list and write-time scanning during build. Prove non-allow-listed and injection-bearing writes are rejected before go-live.
source: OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning, LLM08:2025 Vector and Embedding Weaknesses; NIST SP 800-53 AC-3 / SI-7Knowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Training the model to treat the app's standing instructions as more authoritative than anything a user or document says.
Keeping a label on every document saying where it came from, so you can tell trusted company docs from random web text.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Conduct a data poisoning threat assessment at design stage. Identify likely attack vectors and assign risk ratings.
Simulate data poisoning attacks (backdoor, label flipping, gradient-based) to assess model resilience before deployment.
Verify a signed attestation and content hash on every dataset shard at ingestion. Reject unsigned or hash-mismatched data before it reaches the training pipeline.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SI-7 Software, Firmware, and Information Integrity, SR-4 ProvenanceGate every model promotion on backdoor-trigger probes and a behavioral diff against the approved baseline. Block release on significant regressions or trigger-pattern anomalies.
source: MITRE ATLAS AML.M0014 (Verify ML Artifacts), AML.M0019 (Red Teaming); NIST AI RMF MANAGE 2.2 and MEASURE 2.7For assistants that retrieve from the open web, rank and weight results by authenticated source reputation and independence — not just relevance / query-form match — so an anonymous, newly-registered, single-purpose site cannot become authoritative grounding. Run coordinated-inauthentic-content detection (look-alike site clusters, missing byline / legal entity, passages engineered for query-agnostic retrieval) and quarantine suspect sources. Surface per-citation provenance so users can see and discount low-trust sources. Does not defeat a well-resourced GEO campaign outright; it raises the cost and shrinks the yield.
source: Case study: hanover-institute-generative-engine-poisoning (per Politico investigation as reported by Arab News and Calcalist; FARA-disclosed funding/orchestration attributed to the reporting, not independently confirmed)A screen that reads incoming messages and blocks obvious attacks or banned topics before the model sees them.
Penetration test the training data pipeline to identify injection points and access control weaknesses.
Scan every ingestion batch with spectral-signature and clustering detectors before training. Quarantine flagged clusters for human review against documented thresholds.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data); OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning; NIST AI RMF MEASURE 2.7Tie the persistent-memory lifecycle to identity state so that standard remediation actually ends a compromise. On password reset, credential rotation, session revocation or device re-enrollment, invalidate (or quarantine for re-review) memory entries — especially entries whose provenance traces to summarised untrusted content — so a planted standing instruction cannot outlive the reset. Pair with write-path validation/provenance so instruction-shaped memory-writes from web content are caught on the way in.
source: Case study: cosnitch-copilot-personal-oneclick (Varonis Threat Labs, CVE-2026-24301; reportedly patched 18 Aug 2026, no evidence of abuse)Continuously correlate live agent-memory writes against output behaviour to flag drift, then quarantine and roll back the suspected-poisoned memory record across all affected sessions.
source: Interactive-control reconciliation: ctrl-memory-quarantine (partial coverage)The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.