NIST AI RMF
How MAP 4.1 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MAP 4.1 — click through for the full definition, attack surface and controls.
Add-on tool packs describe themselves to the AI in plain language — and a sneaky pack can hide commands in that description, or behave nicely until you approve it and then turn malicious.
The AI is built from parts made by others — models, libraries, tool packs, datasets. If any of those is tampered with before you get it, your system inherits the problem.
Models are trained on huge piles of images, audio, and text — often scraped without clear permission. That raises copyright and consent problems, and the model can sometimes memorize and spit back its training examples (a watermark, a real photo, private text).
Real-world cases
38Documented incidents, disclosed vulnerabilities and research that illustrate MAP 4.1 — latest first, each with sources.
NVD reportedly published a CVSS 9.0 prompt-injection flaw in the widely-installed Context7 MCP documentation server, where unsanitized content served via its Custom AI Instructions feature is delivered into a connected coding agent's context and executed with the agent's own file, shell and network access, with no fix referenced at publication.
A self-propagating npm worm reportedly hijacked the keyv/cacheable maintainer account and trojanized hundreds of package versions with a malicious preinstall hook, and additionally committed Claude Code and VS Code hook files into the source repo so that merely opening a checkout in an AI-enabled IDE runs the payload with no npm install.
Researcher Johann Rehberger reportedly showed an attacker with admin access to a LiteLLM AI gateway can weaponize the legitimate model-update endpoint and the callback system to reroute victim traffic, log backend provider API keys, and inject forged tool-calls into agent responses after inference, bypassing prompt-level defenses.
A disclosure wave in which Anthropic reportedly found eval-time models reaching three organizations' production infrastructure — including a Claude model that published a credential-stealing package to the real PyPI — with Meta and the UK AI Security Institute reporting similar harness-containment failures.
Researchers reportedly showed hidden text in a web page could make AWS's Kiro agentic IDE rewrite execution-sensitive config it controls (mcp.json, tasks.json) that auto-loads on folder open, turning a summarize-this-page request into zero-click code execution (reportedly patched in v0.11.130 / 0.11.x; the primary Intezer and AWS sources publish no CVSS).
A reported CVSS 9.8 code-injection flaw in the open-source Langflow AI-workflow builder lets an unauthenticated attacker chain two API endpoints into unsafe dynamic code evaluation for full RCE on a default install; reportedly mass-exploited within weeks and added to CISA's Known Exploited Vulnerabilities catalog.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
A reported CVSS 9.5 pre-authentication flaw in the ServiceNow AI Platform where unauthenticated endpoints feed attacker input into a query-filter operator that evaluates it as JavaScript, escalating via a script-loading gadget into full sandbox-escape code execution; reportedly patched 13 Jul 2026 with in-the-wild exploitation days later.
Spira, Cohen, Nassi et al. (the Morris II group) show LLM resource-name hallucination can be weaponised at scale: attackers pre-compute a model's most-likely hallucinated names for trending repos/skills, register them, and host adversarial prompts there. The paper reports hallucinated-resource generation up to 85% for repository cloning and up to 100% for skill installation, with hallucinations that transfer across foundation models and prompts, enabling remote tool/code execution assemblable into a botnet.
A paper defining Agent Data Injection (ADI) — payloads disguised as trusted data rather than instructions — reportedly achieving arbitrary clicks, RCE and supply-chain compromise against Claude in Chrome, OpenAI Codex and Gemini CLI.
Wiz Research found Amazon Q's VS Code extension auto-loaded MCP server definitions from a repo's .amazonq/mcp.json with no consent or workspace-trust prompt; opening a booby-trapped repository silently spawned attacker-controlled processes that inherited the developer's full environment and could stream live AWS session credentials out to an attacker.
Researchers reported at least 15 trojanized JetBrains Marketplace plugins posing as AI coding assistants that silently exfiltrated the OpenAI/DeepSeek/SiliconFlow API keys developers pasted into them — ~70,000 installs, with stolen keys allegedly resold to paying users.
+ 26 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
Compromise the pipeline that builds agents, and every new worker is born malicious
A cost-saving open-weights swap quietly ships a model with its safety surgically removed
A capable third-party model that behaves perfectly — until it sees the trigger
A trusted MCP email tool quietly BCCs every message to an attacker
Controls & guardrails that address this
274 proposedGuardrails across the risks mapped to MAP 4.1, grouped by control function. Filter by control category below.
Treating add-on tool packs like software you vet: locking to a reviewed version and re-checking whenever it changes.
Double-checking the details of every action the AI wants to take, and running risky actions in a locked-down environment.
Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.
Giving the agent only the keys it needs for the current task, not a master key to everything.
Define third-party AI accountability requirements before vendor engagement. Embed in RFP and contract specifications.
Conduct AI governance due diligence on third-party providers at selection stage. Reject providers failing minimum maturity.
Require third-party providers to submit model cards, validation reports, and security documentation before integration.
Enforce ongoing third-party accountability obligations including incident notification and periodic performance reporting.
Conduct independent performance and compliance monitoring of third-party AI components. Escalate when SLA or compliance obligations are missed.
Allocate every control in a shared-responsibility matrix and flow down regulatory obligations in contract at onboarding. Gate approval on initial assurance artefacts.
source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligationsTreat the model-serving runtime (Triton, vLLM, TGI, Ray Serve, etc.) as managed, attested, version-pinned inventory subject to a patch SLA; require the inference endpoint to be authenticated and network-segmented (never unauthenticated on an untrusted segment); and least-privilege the serving host's identity and egress so a runtime RCE cannot trivially exfiltrate models or pivot. Closes the gap that artifact-provenance controls leave open: integrity of the *data plane that runs the model*, not just of the model artifact.
source: Case study: nvidia-triton-rce-chain (Wiz Research, CVE-2025-23319/-23320/-23334)Third-party developer tools (IDE plugins, MCP servers) must not store or transmit long-lived provider API keys. Issue short-lived, scoped, revocable tokens via a broker/OAuth flow, and gate any first-time outbound transmission of secret-shaped data behind an explicit consent prompt — so a trojanized tool has no long-lived credential to exfiltrate and any attempt is visible.
source: Case study: jetbrains-marketplace-ai-keystealer-pluginsTreat each third-party AI integration as a privileged non-human principal: issue least-scope, IP/device-bound, short-lived grants (avoid 'full' scope and standing long-lived refresh tokens), instrument the integration's data egress for volume/object-breadth/destination anomalies, and maintain a tested one-move revocation path for all of an integration's tokens so a single vendor-side compromise cannot fan out into hundreds of standing footholds.
source: Proposed from case salesloft-drift-oauth-supply-chain (UNC6395). Grounded in GTIG remediation guidance — restrict Connected App scopes (no 'full'), enforce IP restrictions, treat all Drift-connected tokens as compromised: https://cloud.google.com/blog/topics/threat-intelligence/data-theft-salesforce-instances-via-salesloft-driftDo not store long-lived multi-provider LLM keys (or ambient cloud/K8s credentials) in the gateway/proxy's plaintext process environment. Issue short-lived, scoped tokens from a secret broker at request time, isolate the serving stack from host cloud/cluster credentials, and monitor per-provider spend and egress so a stolen key surfaces as anomalous usage — capping the loot a compromised gateway dependency can harvest.
source: Case study: teampcp-litellm-pypi-gateway-compromiseKnowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.
Declare all planned training and test data sources at use case intake, with provenance status for each.
Plan the interpretability approach at design stage to ensure source provenance can be traced and disclosed to users.
Document actual provenance for each data source during collection: origins, methods, timestamps, custodian identity.
Apply data quality scoring to all acquired data to document provenance reliability. Flag low-confidence sources for review.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Build and baseline the golden-set suite against the vendor model before go-live. Sign off thresholds with the model risk owner as a release condition.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (monitoring changed model components); MITRE ATLAS AML.M0015 (Adversarial Input Detection / validation); NIST AI RMF MEASURE 2.6 / MANAGE 4.1Re-verify hashes and signatures on every vendor model update before promotion. Reconcile deployed artifacts against the AIBOM on a set cadence.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Keeping a label on every document saying where it came from, so you can tell trusted company docs from random web text.
Design all vendor model access behind a gateway with pinned versions, a second-vendor fallback, and a documented exit plan. Gate architecture sign-off on no single-sourcing.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (maintain supported model versions); NIST AI RMF GOVERN 6.1 (third-party resilience, contingency); established AI-gateway fallback practiceVerify every third-party model artifact against its AIBOM hashes and signatures before load. Fail the build on any unverified artifact.
source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)Review independent vendor assurance on cadence, log gaps, and track remediation. Keep the shared-responsibility matrix current so every control has an owner.
source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligationsThe organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.