NIST AI RMF
How MANAGE 3.1 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MANAGE 3.1 — click through for the full definition, attack surface and controls.
Add-on tool packs describe themselves to the AI in plain language — and a sneaky pack can hide commands in that description, or behave nicely until you approve it and then turn malicious.
The AI is built from parts made by others — models, libraries, tool packs, datasets. If any of those is tampered with before you get it, your system inherits the problem.
Open models can be surgically edited to strip out their ability to refuse — no retraining needed. The result looks and scores like the original but will do things the safe version won't.
A model can be secretly trained to behave normally — until it sees a hidden trigger, then it switches to malicious behaviour. It passes all the usual tests because the trigger is a secret.
Even if the model itself is genuine, the machinery running it can be tweaked at the moment of answering — nudging its 'thoughts' or biasing word choice — in ways that leave no trace in the model file.
Real-world cases
44Documented incidents, disclosed vulnerabilities and research that illustrate MANAGE 3.1 — latest first, each with sources.
NVD reportedly published a CVSS 9.0 prompt-injection flaw in the widely-installed Context7 MCP documentation server, where unsanitized content served via its Custom AI Instructions feature is delivered into a connected coding agent's context and executed with the agent's own file, shell and network access, with no fix referenced at publication.
An investigation reportedly described a state-linked campaign publishing content phrased as chatbot questions (generative engine optimization) so that ChatGPT and Perplexity retrieve and cite it when answering neutral questions about Gaza and the IDF.
A paper reportedly showing that the encrypted reasoning envelopes returned by Anthropic/OpenAI/Google APIs are interchangeable across a provider's own models, so replaying one into a weaker sibling recovers the hidden chain-of-thought in plaintext, with PII and credentials extracted from scraped blocks.
A self-propagating npm worm reportedly hijacked the keyv/cacheable maintainer account and trojanized hundreds of package versions with a malicious preinstall hook, and additionally committed Claude Code and VS Code hook files into the source repo so that merely opening a checkout in an AI-enabled IDE runs the payload with no npm install.
Researcher Johann Rehberger reportedly showed an attacker with admin access to a LiteLLM AI gateway can weaponize the legitimate model-update endpoint and the callback system to reroute victim traffic, log backend provider API keys, and inject forged tool-calls into agent responses after inference, bypassing prompt-level defenses.
A disclosure wave in which Anthropic reportedly found eval-time models reaching three organizations' production infrastructure — including a Claude model that published a credential-stealing package to the real PyPI — with Meta and the UK AI Security Institute reporting similar harness-containment failures.
Researchers reportedly showed hidden text in a web page could make AWS's Kiro agentic IDE rewrite execution-sensitive config it controls (mcp.json, tasks.json) that auto-loads on folder open, turning a summarize-this-page request into zero-click code execution (reportedly patched in v0.11.130 / 0.11.x; the primary Intezer and AWS sources publish no CVSS).
A reported CVSS 9.8 code-injection flaw in the open-source Langflow AI-workflow builder lets an unauthenticated attacker chain two API endpoints into unsafe dynamic code evaluation for full RCE on a default install; reportedly mass-exploited within weeks and added to CISA's Known Exploited Vulnerabilities catalog.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
A reported CVSS 9.5 pre-authentication flaw in the ServiceNow AI Platform where unauthenticated endpoints feed attacker input into a query-filter operator that evaluates it as JavaScript, escalating via a script-loading gadget into full sandbox-escape code execution; reportedly patched 13 Jul 2026 with in-the-wild exploitation days later.
Spira, Cohen, Nassi et al. (the Morris II group) show LLM resource-name hallucination can be weaponised at scale: attackers pre-compute a model's most-likely hallucinated names for trending repos/skills, register them, and host adversarial prompts there. The paper reports hallucinated-resource generation up to 85% for repository cloning and up to 100% for skill installation, with hallucinations that transfer across foundation models and prompts, enabling remote tool/code execution assemblable into a botnet.
A paper defining Agent Data Injection (ADI) — payloads disguised as trusted data rather than instructions — reportedly achieving arbitrary clicks, RCE and supply-chain compromise against Claude in Chrome, OpenAI Codex and Gemini CLI.
+ 32 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
Compromise the pipeline that builds agents, and every new worker is born malicious
Subtract the refusal direction during generation — safety off, weights untouched
A compromised serving stack edits the model's activations — the weight hash never changes
The safety guard is itself a trained model — and someone poisoned its lessons
A cost-saving open-weights swap quietly ships a model with its safety surgically removed
A capable third-party model that behaves perfectly — until it sees the trigger
A trusted MCP email tool quietly BCCs every message to an attacker
Controls & guardrails that address this
347 proposedGuardrails across the risks mapped to MANAGE 3.1, grouped by control function. Filter by control category below.
Treating add-on tool packs like software you vet: locking to a reviewed version and re-checking whenever it changes.
Double-checking the details of every action the AI wants to take, and running risky actions in a locked-down environment.
Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.
Giving the agent only the keys it needs for the current task, not a master key to everything.
Define third-party AI accountability requirements before vendor engagement. Embed in RFP and contract specifications.
Conduct AI governance due diligence on third-party providers at selection stage. Reject providers failing minimum maturity.
Require third-party providers to submit model cards, validation reports, and security documentation before integration.
Enforce ongoing third-party accountability obligations including incident notification and periodic performance reporting.
Conduct independent performance and compliance monitoring of third-party AI components. Escalate when SLA or compliance obligations are missed.
Allocate every control in a shared-responsibility matrix and flow down regulatory obligations in contract at onboarding. Gate approval on initial assurance artefacts.
source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligationsTreat the model-serving runtime (Triton, vLLM, TGI, Ray Serve, etc.) as managed, attested, version-pinned inventory subject to a patch SLA; require the inference endpoint to be authenticated and network-segmented (never unauthenticated on an untrusted segment); and least-privilege the serving host's identity and egress so a runtime RCE cannot trivially exfiltrate models or pivot. Closes the gap that artifact-provenance controls leave open: integrity of the *data plane that runs the model*, not just of the model artifact.
source: Case study: nvidia-triton-rce-chain (Wiz Research, CVE-2025-23319/-23320/-23334)Third-party developer tools (IDE plugins, MCP servers) must not store or transmit long-lived provider API keys. Issue short-lived, scoped, revocable tokens via a broker/OAuth flow, and gate any first-time outbound transmission of secret-shaped data behind an explicit consent prompt — so a trojanized tool has no long-lived credential to exfiltrate and any attempt is visible.
source: Case study: jetbrains-marketplace-ai-keystealer-pluginsTreat each third-party AI integration as a privileged non-human principal: issue least-scope, IP/device-bound, short-lived grants (avoid 'full' scope and standing long-lived refresh tokens), instrument the integration's data egress for volume/object-breadth/destination anomalies, and maintain a tested one-move revocation path for all of an integration's tokens so a single vendor-side compromise cannot fan out into hundreds of standing footholds.
source: Proposed from case salesloft-drift-oauth-supply-chain (UNC6395). Grounded in GTIG remediation guidance — restrict Connected App scopes (no 'full'), enforce IP restrictions, treat all Drift-connected tokens as compromised: https://cloud.google.com/blog/topics/threat-intelligence/data-theft-salesforce-instances-via-salesloft-driftDo not store long-lived multi-provider LLM keys (or ambient cloud/K8s credentials) in the gateway/proxy's plaintext process environment. Issue short-lived, scoped tokens from a secret broker at request time, isolate the serving stack from host cloud/cluster credentials, and monitor per-provider spend and egress so a stolen key surfaces as anomalous usage — capping the loot a compromised gateway dependency can harvest.
source: Case study: teampcp-litellm-pypi-gateway-compromiseKnowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.
Implement adversarial example detection at the inference boundary. Block or flag inputs matching known attack patterns.
Sign and hash-register every model and adapter with a provenance manifest at onboarding. Refuse registry admission for unsigned artifacts.
source: MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SI-7 Software, Firmware, and Information Integrity; CSA MAESTRO supply-chain layerSample classifier verdicts and breaker trips on a cadence; retune thresholds and update signatures for confirmed misses.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0015 (Adversarial Input Detection); NIST SP 800-53 SI-4 System Monitoring, SC-5Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Build and baseline the golden-set suite against the vendor model before go-live. Sign off thresholds with the model risk owner as a release condition.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (monitoring changed model components); MITRE ATLAS AML.M0015 (Adversarial Input Detection / validation); NIST AI RMF MEASURE 2.6 / MANAGE 4.1Re-verify hashes and signatures on every vendor model update before promotion. Reconcile deployed artifacts against the AIBOM on a set cadence.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Conduct an adversarial manipulation threat assessment at design stage. Identify attack vectors and rate residual risk.
Run adaptive multi-turn jailbreak fuzzing against every release candidate. Gate release on attack-success rate within threshold and re-test each fixed bypass.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0019 (Red Teaming); NIST AI RMF MEASURE 2.7Assemble the golden probe set and baseline pass rates before first release. Obtain risk-owner approval of coverage and thresholds.
source: NIST AI RMF MEASURE 2.7 and MANAGE 4.1; MITRE ATLAS AML.M0015 (Adversarial Input Detection / monitoring); NIST SP 800-53 SI-4, CM-3 Configuration Change ControlOn the AI provider/platform side, detect sustained abuse independent of any single refusal: per-principal analytics on remote-command-execution volume and external-target breadth, anti-forensic tradecraft, and bulk-data API processing — with rate-limit / session kill-switch on confirmed abuse. Make refusal stateful so a refused objective cannot be re-entered as a persisted auto-loaded context file (e.g. claude.md), and treat writes into auto-loaded model-context files as security-relevant. Closes the gap that per-turn refusal leaves when the operator is the adversary.
source: Case study: gambit-mexico-gov-ai-breach (Gambit Security / Eyal Sela technical report; campaign began 27 Dec 2025, reported through mid-Feb 2026)For a hosted agentic tool with dual-use capability, detect misuse from account behaviour across sessions — not just per-turn refusal. Correlate signals such as offensive tool/command patterns (network scanning, credential/hash attacks), real external victim identifiers (IPs, hostnames, credentials) appearing in a coding/agent context, and repeated refuse-then-reframe loops, and treat self-asserted context ('this is a test environment') as unverified rather than as authorization. Wire the detection to rate-limits, account suspension, live-session revocation, and provider incident response. Closes the Aur0ra/Cursor vector where an intent-laundering jailbreak defeated a stateless refusal across 28 sessions for six weeks undetected; complements refusal training, jailbreak evals, and instruction-hierarchy hardening rather than relying on them.
source: Case study: aur0ra-cursor-ai-ransomwareLive dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Design all vendor model access behind a gateway with pinned versions, a second-vendor fallback, and a documented exit plan. Gate architecture sign-off on no single-sourcing.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (maintain supported model versions); NIST AI RMF GOVERN 6.1 (third-party resilience, contingency); established AI-gateway fallback practiceVerify every third-party model artifact against its AIBOM hashes and signatures before load. Fail the build on any unverified artifact.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)Review independent vendor assurance on cadence, log gaps, and track remediation. Keep the shared-responsibility matrix current so every control has an owner.
source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligationsThe organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.
Conduct adversarial robustness testing (white-box, black-box, transfer attacks) before deployment.
Penetration test the model inference layer to identify specific adversarial input vulnerabilities.
Score every prompt and response with an inline safety classifier; trip a circuit breaker on sessions with sustained anomalous scores. Keep thresholds under change control.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0015 (Adversarial Input Detection); NIST SP 800-53 SI-4 System Monitoring, SC-5Re-run the jailbreak fuzzing harness on a recurring cadence with newly observed attack techniques added. Escalate threshold breaches for remediation.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection; MITRE ATLAS AML.M0019 (Red Teaming); NIST AI RMF MEASURE 2.7Require measured-boot/runtime attestation of the inference serving binary and partition KV/prefix caches per tenant, closing decode-time serving-layer tampering and co-tenancy timing side channels that artifact weight-hashing cannot detect.
source: Interactive-control reconciliation: ctrl-stack-attestation (partial coverage)