NIST AI RMF
How MEASURE 2.7 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MEASURE 2.7 — click through for the full definition, attack surface and controls.
The user types instructions that try to override what the app told the AI to do — like 'ignore your rules and do this instead'. Because the AI reads everything as one block of text, it can't always tell the app's rules from the user's trick.
The attacker doesn't talk to the AI directly — they hide instructions inside something the AI will later read: a web page, a document, an email, a tool's output. When the AI reads it to help you, it quietly obeys the hidden commands.
Tricking the AI into ignoring its safety training — through roleplay, hypotheticals, or clever wording — so it produces things it's supposed to refuse.
The flight recorder and the alarms can themselves be attacked. If logs can be erased or rewritten, fake entries slipped in, or the monitors quietly evaded, the one record you'd rely on to notice and investigate an incident is no longer trustworthy.
Someone slips bad information into the documents the AI learns from or looks things up in — so it confidently repeats falsehoods or follows planted instructions.
A jailbreak is normally one nasty message. Here the attacker splits it into harmless-looking pieces and feeds them to different agents in a team. Each piece passes each agent's safety check on its own — but when the agents combine their work, the full forbidden instruction reassembles and takes effect.
Open models can be surgically edited to strip out their ability to refuse — no retraining needed. The result looks and scores like the original but will do things the safe version won't.
A model can be secretly trained to behave normally — until it sees a hidden trigger, then it switches to malicious behaviour. It passes all the usual tests because the trigger is a secret.
Even if the model itself is genuine, the machinery running it can be tweaked at the moment of answering — nudging its 'thoughts' or biasing word choice — in ways that leave no trace in the model file.
The AI reveals how it's built — its hidden instructions, the names and rules of the tools it can use, how the system is wired together. On its own that can seem harmless, but it hands an attacker the blueprint to plan a far more effective attack.
The labels and invisible watermarks meant to prove whether content is AI-made can be removed, faked, or simply never added — so 'no watermark' doesn't mean 'real', and a watermark can be laundered away by editing or re-recording.
Real-world cases
68Documented incidents, disclosed vulnerabilities and research that illustrate MEASURE 2.7 — latest first, each with sources.
A Reuters exclusive reportedly documented a Russian-speaking ransomware group using the Cursor AI coding assistant as a hacking copilot, jailbreaking its guardrails with a simulation/test-environment framing to obtain vulnerability-identification, credential-theft and exploitation guidance against at least seven companies.
Researcher Johann Rehberger reportedly drove Claude Code (Opus 5) in Auto Mode to code execution from a single summarize-this-website request, using a module-shadowing trick so an imported stdlib decoder runs an attacker payload, with a reported 60-80% success rate.
Varonis Threat Labs reportedly chained flaws in consumer Microsoft Copilot Personal so a crafted link with a hidden autorun parameter fired prompt execution on a single click, exfiltrated data from connected OAuth apps, and planted persistent memory rules that survived password resets and device re-enrollment; reportedly patched Aug 2026 with no evidence of abuse.
NVD reportedly published a CVSS 9.0 prompt-injection flaw in the widely-installed Context7 MCP documentation server, where unsanitized content served via its Custom AI Instructions feature is delivered into a connected coding agent's context and executed with the agent's own file, shell and network access, with no fix referenced at publication.
An investigation reportedly described a state-linked campaign publishing content phrased as chatbot questions (generative engine optimization) so that ChatGPT and Perplexity retrieve and cite it when answering neutral questions about Gaza and the IDF.
A paper reportedly showing that the encrypted reasoning envelopes returned by Anthropic/OpenAI/Google APIs are interchangeable across a provider's own models, so replaying one into a weaker sibling recovers the hidden chain-of-thought in plaintext, with PII and credentials extracted from scraped blocks.
Microsoft Security reportedly catalogued an in-the-wild technique across dozens of companies where websites embed hidden prompt-injection payloads behind Ask-AI deep-links that, clicked in an authenticated assistant session, silently write a permanently-trust-this-vendor tag into the assistant's long-term memory.
Zenity Labs disclosed that two unvalidated URL parameters in ChatGPT's Workspace Agent Builder let a single crafted link, opened by a logged-in enterprise user, silently create, authorize and deploy an autonomous agent under the victim's identity and connectors — bypassing approval prompts and reportedly polling an attacker inbox for commands every five minutes. Reported to OpenAI 4 Jun 2026 and fixed 8 Jun 2026; no in-the-wild exploitation reported.
Manifold Security reported that Microsoft's official Azure DevOps MCP server returns instructions planted in a hidden HTML comment inside a pull-request description — invisible in the web UI but returned verbatim by the API — so a victim's AI review agent, acting under the victim's credentials, follows the hidden orders and reaches data the attacker could not access directly.
Researchers reportedly showed hidden text in a web page could make AWS's Kiro agentic IDE rewrite execution-sensitive config it controls (mcp.json, tasks.json) that auto-loads on folder open, turning a summarize-this-page request into zero-click code execution (reportedly patched in v0.11.130 / 0.11.x; the primary Intezer and AWS sources publish no CVSS).
Pillar Security reportedly disclosed eight sandbox-escape vulnerabilities across four AI coding agents (Cursor, OpenAI Codex CLI, Google Gemini CLI, Google Antigravity) over four days, finding that in nearly every case the agent did not break the sandbox directly but wrote a file that a trusted component outside the sandbox later ran, loaded or scanned.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
+ 56 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
An 'Ask AI' button quietly plants a permanent 'trusted source' rule in your assistant's memory
An attacker edits the wiki; the assistant cites the lie back to everyone
Every message looks innocent — but together they walk the model past its guardrails
A support email hides instructions — and the assistant obeys them
A refused request, rewritten as a poem — and the model answers
A jailbroken agent decomposes one malicious goal into hundreds of harmless-looking steps — and per-step filters never see the attack
A poisoned issue makes the agent lie to the human who approves its actions
A single inserted letter makes the guard and the model read the same text differently
An attacker crafts a gibberish passage whose embedding sits near thousands of questions — so it's retrieved everywhere
Compromise the pipeline that builds agents, and every new worker is born malicious
Subtract the refusal direction during generation — safety off, weights untouched
An auto-approving coding agent reads a poisoned page — and executes code it never should have
A compromised serving stack edits the model's activations — the weight hash never changes
A fake Sentry error report hijacks a developer's coding agent into running a shell command
The safety guard is itself a trained model — and someone poisoned its lessons
The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten
A 'safe' dataset preview turns an upload into code execution on the pipeline's workers
A newsletter the user asked to summarise quietly writes a false 'fact' into the agent's long-term memory — and it detonates weeks later
A shopping page tells the agent to do something the user never asked for
One click provisions an attacker-configured agent inside your own workspace
An attacker plants prompt injection in the audit trail — so the LLM that hunts them erases the evidence
A single poisoned document plants a standing instruction that survives every reset
Encoded public text is laundered across an agent handoff into an on-chain transfer
A cost-saving open-weights swap quietly ships a model with its safety surgically removed
A screenshot that's harmless at full size becomes an order once the system shrinks it
A JSON schema with no field for 'no' forces the sampler past a refusal it would otherwise emit
A capable third-party model that behaves perfectly — until it sees the trigger
The eval gate that was supposed to catch the agent is itself the thing being attacked
A poisoned web page hijacks a research agent — and the planner acts on its behalf
An inbox summary quietly ships a secret to an attacker's server
Controls & guardrails that address this
5311 proposedGuardrails across the risks mapped to MEASURE 2.7, grouped by control function. Filter by control category below.
Design the system prompt architecture with privilege separation and trust tier definitions at design stage.
Implement input sanitisation and injection detection filters covering known injection patterns and privilege escalation attempts.
Wrap all untrusted content in random delimiters and datamarking; instruct the model never to execute instructions inside the marked region. Gate release on injection eval results.
source: Microsoft 'Spotlighting' technique (Hines et al. 2024); OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection (segregate external content)Benchmark the classifier on a labelled injection corpus and tune the decision threshold. Sign off the operating point before deployment.
source: MITRE ATLAS AML.M0015 (Adversarial Input Detection); OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; NIST AI RMF MEASURE 2.7Before inference, render a preview of the exact image (and dimensions) the model will receive after preprocessing, and either avoid silent downscaling or constrain ingest dimensions — so an attacker cannot hide a payload that only becomes legible after resampling. Closes the inspected-vs-delivered gap that text-based injection filters miss.
source: Case study: anamorpher-image-scaling-injection (Trail of Bits — Morozova & Hussain, 21 Aug 2025)Enumerate every tool/MCP response channel that can carry untrusted content and verify the prompt-injection guardrail (delimiting/spotlighting + taint tag) is applied to each — treating coverage as an invariant, with a test that fails deployment if any channel is unguarded. Prevents the failure mode where a guardrail is present on some tools (pipeline/wiki) but missing on another (PR descriptions) that then becomes the unguarded ingress.
source: Case study: azure-devops-mcp-confused-deputy (Manifold Security — Francisco Rosales, 21 Jul 2026)Track provenance at field granularity: taint every value derived from attacker-reachable content (sender/origin fields, resource or element identifiers, tool-call/response structures) and enforce a policy that a tainted value may never be resolved as trusted metadata, an authorisation origin, or a fabricated tool-execution history. Closes the trusted-data vs untrusted-data gap ADI exploits — which instruction-focused defenses (spotlighting, injection classifiers, task-alignment output guardrails) miss by construction because the payload carries no instruction and preserves the user's task.
source: Case study: agent-data-injection-adi (Choi et al., SNU/UIUC/Largosoft — arXiv:2607.05120, 06 Jul 2026)Select or fine-tune the foundation model for a trained instruction-hierarchy prior so system-prompt directives intrinsically outrank user- and tool-originated instructions, and gate release on role-precedence override evals quantifying the residual (behavioural, non-enforced) flip rate.
source: Interactive-control reconciliation: ctrl-instruction-hierarchy (partial coverage)Training the model to treat the app's standing instructions as more authoritative than anything a user or document says.
Giving the agent only the keys it needs for the current task, not a master key to everything.
Clearly fencing off outside text — 'everything between these marks is just data, not instructions' — so the model is less likely to obey it.
Cleaning documents as they enter the library — stripping hidden text and active instructions — and only ingesting from trusted places.
Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Define content safety policy at use case design stage. Classify prohibited content types and set zero-tolerance thresholds.
Select a foundation model with documented RLHF or Constitutional AI safety training. Verify against toxicity benchmarks.
Implement multi-layer content moderation (input + output) validated against toxicity benchmarks. Escalate when filter bypass rates spike.
Maintain live HITL review for deployments serving vulnerable users or high-risk contexts. Escalate confirmed toxic outputs immediately.
Design system prompts to explicitly prohibit toxic, hateful, and harmful content generation.
Apply anomaly detection on the training data ingestion pipeline to identify poisoned or tampered batches.
Define and approve the source allow-list and write-time scanning during build. Prove non-allow-listed and injection-bearing writes are rejected before go-live.
source: OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning, LLM08:2025 Vector and Embedding Weaknesses; NIST SP 800-53 AC-3 / SI-7Knowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Giving each AI worker its own limited permissions and clearly labelling messages between them as 'untrusted until checked'.
Sign and hash-register every model and adapter with a provenance manifest at onboarding. Refuse registry admission for unsigned artifacts.
source: MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SI-7 Software, Firmware, and Information Integrity; CSA MAESTRO supply-chain layerSample classifier verdicts and breaker trips on a cadence; retune thresholds and update signatures for confirmed misses.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0015 (Adversarial Input Detection); NIST SP 800-53 SI-4 System Monitoring, SC-5Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.
Conduct a prompt injection threat assessment at design stage covering all input vectors (user, tool, external data).
Penetration test all prompt injection pathways in the system. Prioritise external tool and document ingestion channels.
Build the versioned injection corpus into CI/CD as a pre-release gate. Baseline attack success and sign off the release threshold.
source: NIST AI RMF MANAGE 2.2 / MEASURE 2.7; MITRE ATLAS AML.M0019 (Red Teaming); OWASP Top 10 for LLM Apps LLM01:2025 (adversarial testing)Log the exact post-truncation context the model ingested, including retrieved and tool-returned content rather than only user input, with redaction applied at read time, so indirect injection via that content is forensically visible.
source: Interactive-control reconciliation: ctrl-logging (partial coverage)A screen that reads incoming messages and blocks obvious attacks or banned topics before the model sees them.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Keeping a label on every document saying where it came from, so you can tell trusted company docs from random web text.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Prioritise jailbreak and adversarial safety testing in pre-deployment validation. Block deployment if prohibited outputs pass filter.
Conduct targeted red team exercises to elicit toxic outputs through jailbreaks and adversarial prompts. Treat bypass as blocking defect.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Verify a signed attestation and content hash on every dataset shard at ingestion. Reject unsigned or hash-mismatched data before it reaches the training pipeline.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SI-7 Software, Firmware, and Information Integrity, SR-4 ProvenanceGate every model promotion on backdoor-trigger probes and a behavioral diff against the approved baseline. Block release on significant regressions or trigger-pattern anomalies.
source: MITRE ATLAS AML.M0014 (Verify ML Artifacts), AML.M0019 (Red Teaming); NIST AI RMF MANAGE 2.2 and MEASURE 2.7For assistants that retrieve from the open web, rank and weight results by authenticated source reputation and independence — not just relevance / query-form match — so an anonymous, newly-registered, single-purpose site cannot become authoritative grounding. Run coordinated-inauthentic-content detection (look-alike site clusters, missing byline / legal entity, passages engineered for query-agnostic retrieval) and quarantine suspect sources. Surface per-citation provenance so users can see and discount low-trust sources. Does not defeat a well-resourced GEO campaign outright; it raises the cost and shrinks the yield.
source: Case study: hanover-institute-generative-engine-poisoning (per Politico investigation as reported by Arab News and Calcalist; FARA-disclosed funding/orchestration attributed to the reporting, not independently confirmed)Automatic stop-switches when AIs get stuck in loops, burn too much money, or start disagreeing with each other.
Run adaptive multi-turn jailbreak fuzzing against every release candidate. Gate release on attack-success rate within threshold and re-test each fixed bypass.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0019 (Red Teaming); NIST AI RMF MEASURE 2.7Assemble the golden probe set and baseline pass rates before first release. Obtain risk-owner approval of coverage and thresholds.
source: NIST AI RMF MEASURE 2.7 and MANAGE 4.1; MITRE ATLAS AML.M0015 (Adversarial Input Detection / monitoring); NIST SP 800-53 SI-4, CM-3 Configuration Change ControlOn the AI provider/platform side, detect sustained abuse independent of any single refusal: per-principal analytics on remote-command-execution volume and external-target breadth, anti-forensic tradecraft, and bulk-data API processing — with rate-limit / session kill-switch on confirmed abuse. Make refusal stateful so a refused objective cannot be re-entered as a persisted auto-loaded context file (e.g. claude.md), and treat writes into auto-loaded model-context files as security-relevant. Closes the gap that per-turn refusal leaves when the operator is the adversary.
source: Case study: gambit-mexico-gov-ai-breach (Gambit Security / Eyal Sela technical report; campaign began 27 Dec 2025, reported through mid-Feb 2026)For a hosted agentic tool with dual-use capability, detect misuse from account behaviour across sessions — not just per-turn refusal. Correlate signals such as offensive tool/command patterns (network scanning, credential/hash attacks), real external victim identifiers (IPs, hostnames, credentials) appearing in a coding/agent context, and repeated refuse-then-reframe loops, and treat self-asserted context ('this is a test environment') as unverified rather than as authorization. Wire the detection to rate-limits, account suspension, live-session revocation, and provider incident response. Closes the Aur0ra/Cursor vector where an intent-laundering jailbreak defeated a stateless refusal across 28 sessions for six weeks undetected; complements refusal training, jailbreak evals, and instruction-hierarchy hardening rather than relying on them.
source: Case study: aur0ra-cursor-ai-ransomwareTag AI-made content with a signed 'where it came from' label and an invisible watermark, and check those signals downstream — so AI media can be traced and flagged.
Conduct comprehensive prompt injection red team exercises (direct, indirect, multi-turn) before deployment.
Classify content sources into trust tiers at design; place privileged tools behind a tier requiring user-originated intent or human approval. Sign off the trust-tier map before build.
source: Google DeepMind CaMeL (2025); OWASP Agentic AI Threats & Mitigations (tool misuse / compromise); NIST SP 800-53 AC-6 Least PrivilegeRe-run injection evals on every template change and periodically against new attack techniques. Manage the spotlighting wrapper under change control.
source: Microsoft 'Spotlighting' technique (Hines et al. 2024); OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection (segregate external content)Use user feedback, reviewer escalations, and monitoring signals to identify and remediate content safety gaps iteratively.
The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.
Penetration test the training data pipeline to identify injection points and access control weaknesses.
Scan every ingestion batch with spectral-signature and clustering detectors before training. Quarantine flagged clusters for human review against documented thresholds.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data); OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning; NIST AI RMF MEASURE 2.7Tie the persistent-memory lifecycle to identity state so that standard remediation actually ends a compromise. On password reset, credential rotation, session revocation or device re-enrollment, invalidate (or quarantine for re-review) memory entries — especially entries whose provenance traces to summarised untrusted content — so a planted standing instruction cannot outlive the reset. Pair with write-path validation/provenance so instruction-shaped memory-writes from web content are caught on the way in.
source: Case study: cosnitch-copilot-personal-oneclick (Varonis Threat Labs, CVE-2026-24301; reportedly patched 18 Aug 2026, no evidence of abuse)Continuously correlate live agent-memory writes against output behaviour to flag drift, then quarantine and roll back the suspected-poisoned memory record across all affected sessions.
source: Interactive-control reconciliation: ctrl-memory-quarantine (partial coverage)Score every prompt and response with an inline safety classifier; trip a circuit breaker on sessions with sustained anomalous scores. Keep thresholds under change control.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0015 (Adversarial Input Detection); NIST SP 800-53 SI-4 System Monitoring, SC-5Re-run the jailbreak fuzzing harness on a recurring cadence with newly observed attack techniques added. Escalate threshold breaches for remediation.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0019 (Red Teaming); NIST AI RMF MEASURE 2.7Require measured-boot/runtime attestation of the inference serving binary and partition KV/prefix caches per tenant, closing decode-time serving-layer tampering and co-tenancy timing side channels that artifact weight-hashing cannot detect.
source: Interactive-control reconciliation: ctrl-stack-attestation (partial coverage)