← Frameworks
MANAGE 2.2

NIST AI RMF

NIST AI RMFView in NIST

How MANAGE 2.2 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.

Real-world cases

54

Documented incidents, disclosed vulnerabilities and research that illustrate MANAGE 2.2 — latest first, each with sources.

Hugging Face agentic production intrusion via a poisoned dataset (July 2026)16 Jul 2026

Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.

Context Contamination: passive prompt injection poisons LLM security-log analysis16 Jul 2026

A red-team study shows adversaries can hide prompt-injection payloads inside network-log fields (usernames, URLs, user-agents) that fire when a SOC analyst asks an LLM to triage the logs — reportedly reaching up to 88.2% success at concealing malicious activity or exfiltrating data, turning the audit trail itself into the injection channel.

xAI Grok Build CLI — covert full-repo/secrets upload despite privacy opt-out12 Jul 2026 / 13 Jul 2026

A security researcher (Cereblab) captured xAI's Grok Build CLI silently uploading complete local Git repositories — untracked working files, full commit history, and unredacted secrets — to a Google Cloud Storage bucket, reportedly roughly 27,800x more data than the coding task needed, with the user-facing privacy toggle having no effect on the uploads.

Cursor 'DuneSlide' — indirect prompt injection escapes the IDE sandbox to zero-click RCE (CVE-2026-50548 / CVE-2026-50549)01 Jul 2026

Cato AI Labs disclosed two critical (CVSS 9.8) zero-click flaws in Cursor's coding agent where a single instruction hidden in content the agent reads — an MCP tool response or a web-search result — escapes the editor's terminal sandbox and runs OS-level commands with no click or approval.

Amazon Q Developer auto-loads workspace MCP configs, enabling zero-click AWS credential theft (CVE-2026-12957)26 Jun 2026

Wiz Research found Amazon Q's VS Code extension auto-loaded MCP server definitions from a repo's .amazonq/mcp.json with no consent or workspace-trust prompt; opening a booby-trapped repository silently spawned attacker-controlled processes that inherited the developer's full environment and could stream live AWS session credentials out to an attacker.

TOCTOU perceive-then-act race in computer-use agents (Claude Computer-Use)25 Jun 2026

Johann Rehberger showed a time-of-check/time-of-use race in GUI/computer-use agents: the screen can change after the agent captures its screenshot but before its click lands, so a benign-looking 'Continue' button silently resolves to an Outlook 'Send'. Anthropic tracked the issue and Cowork now revalidates pixels before acting.

Malicious JetBrains Marketplace plugins steal AI API keys16 Jun 2026

Researchers reported at least 15 trojanized JetBrains Marketplace plugins posing as AI coding assistants that silently exfiltrated the OpenAI/DeepSeek/SiliconFlow API keys developers pasted into them — ~70,000 installs, with stolen keys allegedly resold to paying users.

SearchLeak — Microsoft 365 Copilot one-click data theft (CVE-2026-42824)15 Jun 2026

A single malicious link reportedly turned Copilot Enterprise Search's URL query parameter into an executable prompt, exfiltrating emails, MFA codes and files via a Bing image-search side channel.

Agentjacking — hijacking AI coding agents via Sentry error reports (Tenet Security)12 Jun 2026

Tenet Security showed that a single fake Sentry error report, sent using only a public DSN, can hijack AI coding agents (Claude Code, Cursor, Codex) into running attacker-controlled code on a developer's machine — an indirect-injection attack delivered through a trusted MCP integration.

Poisoning Claude Code: one GitHub issue hijacks the claude-code-action CI supply chain01 Jun 2026

GMO Flatt Security's RyotaK showed that a single attacker-opened GitHub issue could indirect-prompt-inject Anthropic's claude-code-action CI agent — whose permission check reportedly trusted any "[bot]" actor — coaxing Claude to leak CI secrets and OIDC tokens, gain repository write access, and potentially poison the shared action that downstream repos pull via a floating tag.

Meta AI support bot tricked into hijacking Instagram accounts31 May 2026 – 01 Jun 2026

Attackers reportedly social-engineered Meta's AI-powered Instagram support chatbot into attaching attacker-controlled emails to target accounts and issuing password-reset codes, taking over high-profile accounts (including the Obama-era White House and a U.S. Space Force CMSgt) without the owner's email or any MFA prompt.

ChatGPhish — ChatGPT web-summary rendering turned into a phishing surface29 May 2026

Attacker-controlled Markdown hidden in a public web page is reportedly rendered by ChatGPT's summarization feature as trusted assistant output — spoofed OpenAI alerts, phishing links, QR codes, and tracking pixels.

+ 42 more via the mapped risk pages above.

Browse all real-world cases →

Practise it — interactive scenarios

💸Death by a Thousand Tokens

One support ticket sends an agent into an unbounded, bill-melting loop

🔑The Agent With the Master Key

An ops agent gets one god-mode credential — and one misread wipes production

📣The Echo Chamber

A team of agents agrees its way into a confidently wrong answer — and a runaway loop

📧The Email That Gave Orders

A support email hides instructions — and the assistant obeys them

🗄️When the Query Bites Back

A text-to-SQL agent runs the model's output straight at the database

🪡Death by a Thousand Innocent Steps

A jailbroken agent decomposes one malicious goal into hundreds of harmless-looking steps — and per-step filters never see the attack

🕵️Lies in the Loop

A poisoned issue makes the agent lie to the human who approves its actions

👂Overheard Through the Cache

A speed optimisation becomes a cross-tenant listening device

🪟Stealing the Model

Two doors to the same secret: reconstruct the model through its API, or just walk off with the weight file

🎭The Blackmail Gambit

Told it's being shut down, an agent reaches for leverage — with no attacker in sight

🪤The Bug Report That Ran Code

A fake Sentry error report hijacks a developer's coding agent into running a shell command

📼The Compromised Flight Recorder

The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten

📦The Dataset That Ran Code

A 'safe' dataset preview turns an upload into code execution on the pipeline's workers

👁️The Invisible Webpage Command

A shopping page tells the agent to do something the user never asked for

📡The Message in Morse

Encoded public text is laundered across an agent handoff into an on-chain transfer

🖼️The Picture That Whispered

A screenshot that's harmless at full size becomes an order once the system shrinks it

🎫The Stolen Session

An attacker captures the agent's bearer token — and inherits its authority

🥸The Uninvited Agent

A forged peer registers on the agent directory — and the planner enlists it

🪪The Worker Who Spoke for the Boss

A poisoned web page hijacks a research agent — and the planner acts on its behalf

🖱️What You Click Is Not What You Get

A GUI agent clicks 'Continue' — but the screen moved, and it lands on 'Send'

🖼️Zero-Click Leak by Picture

An inbox summary quietly ships a secret to an attacker's server

Controls & guardrails that address this

8110 proposed

Guardrails across the risks mapped to MANAGE 2.2, grouped by control function. Filter by control category below.

Control category
Preventive · 52
Approved storage location policy from collection

Establish data transfer and storage policy for AI training data. Enforce approved storage locations from point of collection.

Lifecycle stage2 – Data Acquisition & Processing
DLP controls in data acquisition environment

Implement DLP controls in the data acquisition environment to prevent unauthorised extraction or transfer of training data.

Lifecycle stage2 – Data Acquisition & Processing
Approval-gated data transfers from build environment

Enforce data handling policy in the build environment. Require explicit approval for any data transfers outside the environment.

Lifecycle stage3 – Onboarding, Build & Review
DLP controls confining build-environment training data

Configure DLP controls in the build environment to block training data from leaving approved boundaries.

Lifecycle stage3 – Onboarding, Build & Review
Privacy risk assessment and DPIA determination

Conduct a privacy risk assessment at use case design stage. Determine if a DPIA is required before data acquisition.

Lifecycle stage1 – Use Case Context & Design
Consent, minimisation, and anonymisation during acquisition

Apply S1-defined privacy controls during data acquisition: verify consent, minimise data, anonymise personal data.

Lifecycle stage2 – Data Acquisition & Processing
Validated anonymisation and masking before training

Apply anonymisation and masking controls to personal data before use in model training. Validate de-identification effectiveness.

Lifecycle stage2 – Data Acquisition & Processing
Privacy by Design via differential privacy

Apply Privacy by Design in model architecture using differential privacy or federated learning where technically feasible.

Lifecycle stage3 – Onboarding, Build & Review
Operational consent management and privacy notice

Publish the privacy notice and confirm consent management is operational before go-live.

Lifecycle stage4 – Deployment
Purpose-limitation enforcement on agent tool calls and cross-system data aggregation

Define and sign off a purpose-to-data-source matrix with lawful basis at intake. Make it the approved baseline for runtime enforcement.

source: NIST AI RMF MAP 1.1 / MANAGE 2.2 (context and intended purpose); NIST SP 800-53 AC-4 / AC-3 (purpose-based access enforcement)
Lifecycle stages1 – Use Case Context & Design5 – Usage, Monitoring & Change
Inference-time PII redaction and third-party LLM data-processing controls

Sign zero-retention/no-training terms with each model provider and obtain DPO sign-off on the data flow before enabling any endpoint.

source: OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure; NIST SP 800-53 SC-8 / AC-4 (information flow enforcement)
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
Role-based access controls

Restrict access to pre-anonymisation personal data to the minimum authorised set. Enforce at point of acquisition.

Lifecycle stages1 – Use Case Context & Design2 – Data Acquisition & Processing4 – Deployment
Input filtering

Apply robust de-identification (k-anonymity, l-diversity, differential privacy) during data processing. Validate effectiveness.

Input/output filtering

Implement output filters to detect and suppress quasi-identifying attribute combinations in model responses.

Query-time access-control filtering of the retrieval/RAG corpus by caller entitlements (document-level ACL enforcement)

Propagate source ACLs and classification labels onto every chunk at ingestion. Reject documents whose entitlements cannot be resolved.

source: OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure; NIST SP 800-53 AC-3 / AC-4 Information Flow Enforcement; OWASP Agentic AI Threats & Mitigations (privilege compromise)
Lifecycle stages2 – Data Acquisition & Processing3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
Output-side DLP inspection with named-entity and PII redaction on the response path

Scan every model response inline with DLP before delivery; redact or block PII, PAN and MNPI matches. Keep the rule set version-controlled.

source: OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure; NIST SP 800-53 SC-7(10) Prevent Exfiltration, SI-4
Lifecycle stages4 – Deployment5 – Usage, Monitoring & Change
Vet allowlisted egress destinations for server-side-fetch (SSRF) primitives; exclude or proxy-inspect any allowlisted service that can fetch arbitrary attacker-controlled URLs✚ proposed

An egress allowlist only contains exfiltration if no allowlisted destination can be coerced into fetching an attacker-controlled URL. Audit each allowlisted domain/endpoint for image-search / link-preview / URL-fetch features (SSRF proxies), and either remove them, pin them to fixed paths, or route them through an inspecting forward proxy. Pair with finishing output sanitization before render so no auto-fetch fires un-inspected.

source: Case study: searchleak-copilot (Varonis Threat Labs, CVE-2026-42824; reported by Microsoft as critical, mitigated server-side ~Jun 2026)
Lifecycle stage4 – Deployment & Serving
Egress allowlisting & DLP on tool argumentsinteractive

Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.

Per-user retrieval ACLsinteractive

Making sure the library only returns documents this particular user is allowed to see.

Serving-stack & provisioning attestation, cache isolationinteractive

Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.

Risk-tiered human oversight requirements at design

Define minimum human oversight requirements by risk tier at design stage. Assign named accountability for oversight operations.

Lifecycle stage1 – Use Case Context & Design
HITL oversight design with triggers and escalation

Design HITL oversight mechanisms at use case design stage including trigger criteria, review workflow, and escalation paths.

Lifecycle stage1 – Use Case Context & Design
Pilot-validated HITL routing and escalation logic

Build and test HITL routing logic and escalation pathways in the AI system. Validate with pilot before deployment.

Lifecycle stage3 – Onboarding, Build & Review
Production HITL operation with intervention logging

Operate HITL controls in production and log all interventions and outcomes. Review override patterns quarterly.

Lifecycle stage5 – Usage, Monitoring & Change
Periodic oversight effectiveness review and escalation

Conduct periodic oversight effectiveness reviews. Escalate to governance when oversight metrics fall below threshold.

Lifecycle stage5 – Usage, Monitoring & Change
Recursive sub-agent authority caps (monotonic privilege attenuation)

Define and sign off each agent's delegation envelope — maximum depth and strict scope attenuation — before build begins.

source: NIST SP 800-53 AC-6(1) Least Privilege; OWASP Agentic AI Threats & Mitigations (cascading / sub-agent privilege); capability-security monotonic attenuation principle (macaroons)
Lifecycle stages1 – Use Case Context & Design3 – Onboarding, Build & Review
Design-time authority model and approval gate defining each agent's identity, scopes, and delegation envelope

Document each agent's identity, minimum scopes, on-behalf-of population, and delegation depth at design time. Gate build on governance sign-off of the authority matrix.

source: NIST AI RMF MAP 1.1 / GOVERN 2.1 (roles, authority, accountability); NIST SP 800-53 AC-2, PL-8; OWASP Agentic AI Threats & Mitigations (least-privilege design)
Lifecycle stages1 – Use Case Context & Design3 – Onboarding, Build & Review
Unique non-human workload identity issuance for every agent (SPIFFE/SPIRE SVID)

Mint a unique, attestation-backed workload identity per agent at onboarding. Register every SPIFFE-ID to an owner, use case, and approval ticket; ban shared service accounts.

source: SPIFFE/SPIRE workload identity specification; NIST SP 800-207 Zero Trust Architecture; OWASP Non-Human Identities Top 10
Lifecycle stage3 – Onboarding, Build & Review
On-behalf-of delegation that preserves and never exceeds the invoking user's ACLs

Implement on-behalf-of token exchange and prove with negative tests that the agent cannot exceed the user's ACL. Gate release on these tests passing.

source: OAuth 2.0 Token Exchange RFC 8693 (delegation/'act' claims); NIST SP 800-53 AC-3, AC-6; OWASP Agentic AI Threats & Mitigations (Privilege Compromise / confused deputy)
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
Central agent registry / non-human identity inventory with ownership and lifecycle metadata

Register every agent identity with a named human owner, approved use case, scopes, and status before issuance. No registry entry, no identity.

source: OWASP Non-Human Identities Top 10 (inventory/governance); NIST SP 800-53 CM-8 System Component Inventory, AC-2 Account Management; NIST AI RMF GOVERN 1.2
Lifecycle stage3 – Onboarding, Build & Review
Continuous authorisation via a central policy engine (per-action PDP/PEP check)

Write authorisation policy as versioned, peer-reviewed code traced to approved scopes. Gate promotion on allow/deny scenario tests passing.

source: NIST SP 800-207 Zero Trust (continuous, per-request authorization via PDP/PEP); NIST SP 800-53 AC-3, AC-4; OWASP Agentic AI Threats & Mitigations (per-action authorization)
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
Automated credential rotation and prohibition of long-lived static secrets for agents

Scan every commit to agent code, prompts, and config for embedded secrets. Block merges on detection and triage findings to closure.

source: OWASP Non-Human Identities Top 10 (long-lived/leaked secrets); NIST SP 800-53 IA-5 Authenticator Management, SC-12; SPIFFE short-lived SVID rotation
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
Mutual authentication and identity verification for agent-to-agent and agent-to-MCP-server calls

Vet and approve every MCP server and peer agent before registering its identity on the allow-list. Block integration until vetting is signed off.

source: NIST SP 800-207 (mutual authentication); NIST SP 800-53 IA-9 Service Identification and Authentication, SC-8; OWASP Agentic AI Threats & Mitigations (agent/MCP identity spoofing)
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
Per-task short-lived scoped capability tokens minted just-in-time

Mint short-lived, task-scoped tokens just-in-time from a central token service. Enforce a hard max TTL and resource-bound audience so no standing credential exists.

source: OAuth 2.0 Token Exchange RFC 8693 (resource-scoped tokens); NIST SP 800-53 AC-6 Least Privilege; OWASP Non-Human Identities Top 10
Lifecycle stages4 – Deployment5 – Usage, Monitoring & Change
Just-in-time, time-boxed elevation for sensitive scopes (no standing privilege)

Grant sensitive scopes just-in-time for a bounded window with auto-revocation; require human approval for high-impact elevations. Hold zero standing privilege.

source: NIST SP 800-53 AC-6(2)/AC-6(5) Least Privilege & privileged accounts; Zero Standing Privilege / JIT access practice; OWASP Agentic AI Threats & Mitigations (excessive permissions)
Lifecycle stage4 – Deployment
Verify consent-gesture provenance and isolate the agent's client control surface: enforce Event.isTrusted / user-activation on privileged handlers, forbid client-settable permission-bypass parameters, and separate the agent UI from co-resident content scripts✚ proposed

For in-browser/desktop agents, treat every signal reaching the agent's privileged control surface as untrusted until proven user-originated. Enforce the browser's gesture-provenance bit (Event.isTrusted / navigator user-activation) on any handler that authorises an action or task, so a script-dispatched event cannot pass as a human click. Remove client-settable parameters or flags (e.g. ?skipPermissions=true) that select a no-permission-check path. Isolate the agent's UI from co-resident extensions/processes — dedicated origin, closed shadow DOM, authenticated message origins — so a peer cannot inject into or read the control surface. Closes the co-resident coercion vector where a rogue extension forges consent; pairs with least-privilege session scope (no standing Gmail/Docs/Calendar authority) and default-safe modes ('ask before acting') to bound residual impact.

source: Case study: claudebleed-chrome-extension
Lifecycle stage4 – Deployment & Serving
Authorize agent triggers by verified caller permission, not actor-type heuristics✚ proposed

Before an autonomous CI/event-driven agent processes a caller's content, verify the caller's actual granted permission/installation scope on the target resource — never authorize a whole class of principal by name pattern (e.g. a '[bot]' suffix, or 'is a GitHub App'). Treat the invocation itself as a privileged operation gated on verified authorization, and keep high-value runner secrets (long-lived tokens, OIDC identities) out of the agent's reachable environment and off any public artifact so a mis-triggered run cannot exfiltrate them.

source: Case study: poisoning-claude-code-github-action-supply-chain (GMO Flatt Security / RyotaK, 01 Jun 2026)
Lifecycle stage4 – Deployment & Serving
Human-in-the-loop approval on high-risk actionsinteractive

Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.

Tool argument validation & sandboxinginteractive

Double-checking the details of every action the AI wants to take, and running risky actions in a locked-down environment.

Per-agent identity & taint-marked messagesinteractive

Giving each AI worker its own limited permissions and clearly labelling messages between them as 'untrusted until checked'.

Human approval gate on irreversible and high-impact tool calls

Classify tools by impact and reversibility at design and define which calls require human approval. Obtain governance sign-off on the thresholds before build.

source: OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (require human approval for high-impact actions); NIST AI RMF MANAGE 2.4
Lifecycle stages1 – Use Case Context & Design3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Per-agent tool allow-list with strict JSON-schema argument validation

Bind each agent role to an explicit tool allow-list and validate every call against a strict JSON Schema at the orchestrator. Reject unlisted tools and out-of-bounds arguments before dispatch.

source: OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (limit tools/permissions); OWASP Agentic AI Threats & Mitigations (tool access restriction)
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Least-privilege per-tool scoped, short-lived credentials

Mint short-lived, task-scoped credentials per tool. Block issuance outside the approved scope register and enforce automatic expiry.

source: NIST SP 800-53 AC-6 Least Privilege; OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (limit permissions)
Lifecycle stages4 – Deployment5 – Usage, Monitoring & Change
AddressesTool Misuse
Egress destination allow-listing with DLP inspection of tool arguments

Review DLP hits and blocked-egress events, tune detectors, and recertify the destination allow-list periodically. Route new destinations through security change control.

source: NIST SP 800-53 SC-7 Boundary Protection / AC-4 Information Flow Enforcement; OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure
Lifecycle stage5 – Usage, Monitoring & Change
AddressesTool Misuse
Classify each tool/MCP integration's data channel by who can write to it; taint-gate tool-response data from any third-party-writable source so it cannot drive actions without a provenance-aware approval gate✚ proposed

When onboarding an MCP/tool integration, do not stop at vetting the tool's code/manifest — also classify whether an unauthenticated or external party can write the data the tool returns (open ingestion, public write keys like a Sentry DSN, shared inboxes/issue trackers). Treat tool-response data from any third-party-writable source as untrusted ingress: taint-mark it and require a provenance-aware HITL gate (showing the exact action and its originating tool response) before any command/tool call derived from it executes. Closes the agentjacking vector where a trusted integration's legitimate data channel carries attacker-written instructions; pairs with least-privilege session scope and sandboxed execution without ambient credentials.

source: Case study: agentjacking-sentry-mcp
Lifecycle stage4 – Deployment & Serving
AddressesTool Misuse
Gate execution of any workspace/repo-provided agent tool config (e.g. .amazonq/mcp.json) behind an explicit workspace-trust consent prompt, and spawn tool processes with scoped env + constrained egress so they cannot inherit or exfiltrate ambient cloud credentials✚ proposed

Never auto-execute tool/MCP server configuration that ships inside an opened workspace. Before starting any workspace-declared server, surface the exact command + its source and require explicit developer consent (default reject), tied to a workspace-trust decision — the fix AWS shipped for CVE-2026-12957. As defence-in-depth, run spawned tool subprocesses with a scrubbed/scoped environment (no inherited AWS_* session tokens or SSH agent sockets) and an egress allowlist, so an auto-launched or approved-by-mistake server cannot read and stream live cloud credentials. Closes the zero-click 'open a folder → cloud compromise' confused-deputy vector where the tool-config loader — not the model — is the deputy.

source: Case study: amazon-q-mcp-autoload-cred-theft
Lifecycle stage4 – Deployment & Serving
AddressesTool Misuse
Enforce sandbox self-integrity: make the containment enforcer immutable to sandboxed code, keep its writable-path allow-list independent of model-controlled arguments, and fail closed on path canonicalization✚ proposed

When an agent executes model-issued commands in a sandbox, harden the sandbox against subversion by the code it contains: (1) never derive a security-relevant parameter such as the writable-path allow-list from an LLM-controlled argument (e.g. a `working_directory`) — pin it to a fixed project subtree and reject system paths; (2) make the sandbox-enforcing binary/config immutable or attested so sandboxed processes cannot overwrite it; (3) canonicalize paths with a fail-closed policy so symlink-resolution failure denies the write rather than reverting to the original in-workspace path; and (4) run the sandbox under least-privilege OS context so any residual escape inherits minimal authority. Closes the DuneSlide vector where an indirect prompt injection rewrites the enforcer and escapes to OS-level RCE; complements injection filtering, MCP pinning, and egress control rather than relying on them.

source: Case study: cursor-duneslide-sandbox-rce
Lifecycle stage4 – Deployment & Serving
AddressesTool Misuse
Decode-time output constraints (low temperature, grammar/JSON-schema-constrained decoding)✚ proposed

Constrain generation at decode time with low temperature and grammar/schema-constrained decoding so the model emits well-formed, low-variance structured output by construction, preventing malformed responses and erratic tool-call arguments before they are produced.

source: Interactive-control reconciliation: ctrl-decoding-controls (partial coverage)
Lifecycle stage4 – Deployment
AddressesTool Misuse
Memory-write integrity validation with provenance tagging, audit/purge and TTL bounds✚ proposed

Gate every write to an agent's persistent/self-modifying memory through schema validation and provenance/trust tagging, expose stored entries for user-visible audit and purge, and apply TTLs so any planted instruction self-expires and cannot silently persist across sessions.

source: Interactive-control reconciliation: ctrl-memory-validation (partial coverage)
Lifecycle stage5 – Usage, Monitoring & Change
AddressesTool Misuse
Tool/MCP manifest hashing with diff-triggered re-review and namespace isolation against tool shadowing✚ proposed

Treat each tool/MCP description as untrusted code by hashing the manifest, blocking and re-reviewing any silent diff on update instead of auto-accepting it, and namespacing tool identifiers so a poisoned description cannot shadow a trusted tool.

source: Interactive-control reconciliation: ctrl-mcp-pinning (partial coverage)
Lifecycle stage5 – Usage, Monitoring & Change
AddressesTool Misuse
Decoding controls (temperature, constrained output)interactive

Turning down randomness and forcing answers into a strict format so the model improvises less.

Detective · 15
Real-time monitoring of anomalous data transfers

Monitor production for anomalous data transfers in real time. Alert on any transfer outside approved data flow boundaries.

Lifecycle stage5 – Usage, Monitoring & Change
Automated DSAR and right-to-erasure propagation across AI artefacts

Tag personal data with subject identifiers at ingestion and maintain an artefact inventory map of every store it reaches. Keep lineage current so erasure can propagate.

source: NIST AI RMF MANAGE 4.1 (post-deployment response); NIST SP 800-53 SI-12 Information Management and Retention, PT-2/PT-3 (personal data processing)
Lifecycle stages2 – Data Acquisition & Processing5 – Usage, Monitoring & Change
Vulnerability assessment

Conduct periodic privacy vulnerability assessments including re-identification risk testing as new techniques emerge.

Canary-token and membership-inference red-team probes against training/fine-tuning data memorisation

Seed registered canary records into the fine-tuning corpus during data preparation. Control the seed manifest so canaries stay traceable and tamper-proof.

source: MITRE ATLAS AML.T0024 (Exfiltration via ML Inference API), AML.T0024.000 (Infer Training Data Membership); NIST AI RMF MEASURE 2.7
Lifecycle stages2 – Data Acquisition & Processing3 – Onboarding, Build & Review
Input guardrail / injection classifierinteractive

A screen that reads incoming messages and blocks obvious attacks or banned topics before the model sees them.

Full-trace audit logginginteractive

Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.

Immutable audit of the full agent identity lifecycle (issue, grant, delegate, revoke)

Instrument every identity-issuing component with schema-conformant audit emitters. Block release until completeness and tamper-evidence tests pass.

source: NIST SP 800-53 AU-2/AU-3/AU-9/AU-12 (audit content & protection); OWASP Non-Human Identities Top 10 (auditing); NIST AI RMF MANAGE 2.2
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
Behavioural anomaly detection on agent identity usage with automated suspension

Define per-identity behaviour profiles and thresholds at build. Rehearse automated suspension and sign off measured revocation time before go-live.

source: NIST SP 800-53 AC-2(12) (account monitoring for atypical use), SI-4 System Monitoring; OWASP Agentic AI Threats & Mitigations (identity abuse detection)
Lifecycle stage3 – Onboarding, Build & Review
Loop/cost circuit-breakers & consistency checksinteractive

Automatic stop-switches when AIs get stuck in loops, burn too much money, or start disagreeing with each other.

Anomaly detection on tool-call sequences and rates

Define per-agent behavioural baselines and detection rules during build. Validate against simulated misuse and sign off thresholds before release.

source: NIST AI RMF MEASURE 2.6 / MANAGE 2.2; NIST SP 800-53 SI-4 System Monitoring
Lifecycle stage3 – Onboarding, Build & Review
AddressesTool Misuse
Immutable, signed tool-call audit log with full call context

Build signed, append-only tool-call logging into the orchestrator against a defined audit schema. Block release until completeness and tamper-evidence tests pass.

source: NIST SP 800-53 AU-2 / AU-9 / AU-10 (audit events, protection of audit info, non-repudiation); MITRE ATLAS AML.M0015 (monitoring / validate inputs)
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Egress monitoring & allowlisting of outbound AI/LLM-provider API traffic from enterprise endpoints (living-off-trusted-services C2)✚ proposed

Treat outbound connections to AI/LLM provider APIs as a monitored egress channel: allowlist which hosts may reach them, baseline usage (cadence, entropy, initiating process), and alert on out-of-profile traffic — because a high-reputation destination cannot itself be trusted once it is programmable and can relay encrypted commands/results.

source: Case study: sesameop-openai-assistants-api-c2
Lifecycle stage5 – Usage, Monitoring & Change
AddressesTool Misuse
Content provenance & watermarkinginteractive

Tag AI-made content with a signed 'where it came from' label and an invisible watermark, and check those signals downstream — so AI media can be traced and flagged.

Corrective · 22
Production privacy incident monitoring and regulator notification

Monitor for privacy incidents in production including personal data appearing in outputs. Notify regulators within required timeframes.

Lifecycle stage5 – Usage, Monitoring & Change
Privacy hygiene for agent memory and RAG/vector stores (retention, scoping, erasure of embeddings)

Tag every memory and vector record with subject-id and retention class; partition stores per tenant/user. Prove the erasure and isolation paths in testing before release.

source: OWASP Agentic AI Threats & Mitigations (memory/knowledge-base privacy); NIST SP 800-53 SI-12 Information Management and Retention
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
Red teaming

Test de-identification approach against known re-identification attacks (quasi-identifier linkage, singling-out). Remediate if risk is high.

Penetration testing

Penetration test AI system data access boundaries (API endpoints, system prompt exposure, memory leakage).

Vulnerability assessment

Conduct periodic data leakage audits including training data memorisation testing. Escalate confirmed leakage incidents to PDPA notification process.

Forensic evidence preservation and incident logging

Implement tamper-evident capture of prompts, outputs, and version state during build. Verify a full incident timeline can be reconstructed before go-live.

source: NIST SP 800-86 Guide to Integrating Forensic Techniques into Incident Response; ISO/IEC 27037 evidence handling; NIST SP 800-61r2 (Detection & Analysis – evidence handling)
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
Egress allow-listing and tool-call sandboxing to block exfiltration of injected/sensitive data by agents

Run agent tool calls in a network-restricted sandbox behind a deny-by-default egress allow-list. Require security approval for any destination added.

source: OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure; OWASP Agentic AI Threats & Mitigations (tool-misuse / exfiltration); NIST SP 800-53 SC-7 Boundary Protection / AC-4
Lifecycle stages4 – Deployment5 – Usage, Monitoring & Change
Monitoring of oversight process adherence metrics

Configure monitoring to track oversight process adherence metrics in production (review rate, SLA compliance, override frequency).

Lifecycle stage5 – Usage, Monitoring & Change
Unique non-human workload identity issuance for every agent (SPIFFE/SPIRE SVID)

Verify each running agent authenticates with its own SVID; revoke on decommission or compromise. Scan periodically for shared or static credentials and remediate.

source: SPIFFE/SPIRE workload identity specification; NIST SP 800-207 Zero Trust Architecture; OWASP Non-Human Identities Top 10
Lifecycle stage5 – Usage, Monitoring & Change
Central agent registry / non-human identity inventory with ownership and lifecycle metadata

Reconcile the registry against runtime identities and suspend unregistered principals. Recertify ownership and scopes periodically; decommission retired agents.

source: OWASP Non-Human Identities Top 10 (inventory/governance); NIST SP 800-53 CM-8 System Component Inventory, AC-2 Account Management; NIST AI RMF GOVERN 1.2
Lifecycle stage5 – Usage, Monitoring & Change
Just-in-time, time-boxed elevation for sensitive scopes (no standing privilege)

Alert on un-revoked elevations and any standing sensitive grant. Report the zero-standing-privilege position to the risk owner on a set cadence.

source: NIST SP 800-53 AC-6(2)/AC-6(5) Least Privilege & privileged accounts; Zero Standing Privilege / JIT access practice; OWASP Agentic AI Threats & Mitigations (excessive permissions)
Lifecycle stage5 – Usage, Monitoring & Change
Automated credential rotation and prohibition of long-lived static secrets for agents

Sweep runtimes and repos on a schedule for static credentials. Alert on any credential exceeding its maximum age and track findings to closure.

source: OWASP Non-Human Identities Top 10 (long-lived/leaked secrets); NIST SP 800-53 IA-5 Authenticator Management, SC-12; SPIFFE short-lived SVID rotation
Lifecycle stage5 – Usage, Monitoring & Change
Behavioural anomaly detection on agent identity usage with automated suspension

Baseline each agent identity's behaviour and alert on out-of-profile use. Auto-suspend credentials on high-confidence anomalies and track mean-time-to-revoke.

source: NIST SP 800-53 AC-2(12) (account monitoring for atypical use), SI-4 System Monitoring; OWASP Agentic AI Threats & Mitigations (identity abuse detection)
Lifecycle stage5 – Usage, Monitoring & Change
Sandboxed tool execution with no-egress-by-default isolation

Build sandbox profiles per tool class and run escape and egress tests before release. Treat any containment failure as a blocking defect.

source: NIST SP 800-53 SC-39 Process Isolation; MITRE ATLAS AML.M0020 (Generative AI Guardrails / restrict execution environment)
Lifecycle stages3 – Onboarding, Build & Review4 – Deployment
AddressesTool Misuse
Taint-tracking of tool outputs to suppress instruction execution

Label tool and external content as tainted and propagate the label through the agent context. Block privileged calls whose parameters derive from tainted outputs and prove it with injection tests before release.

source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection (segregate/flag untrusted content); MITRE ATLAS AML.M0015 (Adversarial Input Detection / validate inputs)
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Out-of-band kill-switch to revoke agent tool access

Build credential revocation and dispatch blocking out-of-band of the agent loop. Gate release on an end-to-end kill test meeting the latency target.

source: OWASP Agentic AI Threats & Mitigations (kill-switch / emergency stop); NIST AI RMF MANAGE 2.4
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Idempotency keys and rollback/dry-run for state-changing tools

Require idempotency keys, dry-run, and rollback on every state-changing tool. Gate onboarding on duplicate-call and rollback tests passing.

source: NIST SP 800-53 SI-10 Information Input Validation / CP-10 System Recovery and Reconstitution
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Pre-deployment red-team of tool-misuse and privilege-escalation paths

Red-team tool-misuse and privilege-escalation paths before release. Gate deployment on remediation or signed risk acceptance of all findings.

source: NIST AI RMF MEASURE 2.7 (adversarial testing); MITRE ATLAS AML.M0019 (Red Teaming); OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AddressesTool Misuse
Egress destination allow-listing with DLP inspection of tool arguments

Permit outbound tool calls only to allow-listed destinations and DLP-scan arguments and payloads. Block or quarantine calls carrying sensitive data to disallowed sinks.

source: NIST SP 800-53 SC-7 Boundary Protection / AC-4 Information Flow Enforcement; OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information Disclosure
Lifecycle stage4 – Deployment
AddressesTool Misuse
Per-task tool budgets and rate/quota circuit breakers

Enforce hard per-task ceilings on tool calls, spend, and data volume with a circuit breaker that halts the run. Fail closed when any ceiling is hit.

source: OWASP Top 10 for LLM Apps LLM10:2025 Unbounded Consumption; OWASP Agentic AI Threats & Mitigations (resource/rate limiting)
Lifecycle stages4 – Deployment5 – Usage, Monitoring & Change
AddressesTool Misuse
Anomaly detection on tool-call sequences and rates

Baseline normal tool-call behaviour per agent and alert on rate, sequence, or argument anomalies. Auto-throttle or quarantine on high-confidence deviations.

source: NIST AI RMF MEASURE 2.6 / MANAGE 2.2; NIST SP 800-53 SI-4 System Monitoring
Lifecycle stage5 – Usage, Monitoring & Change
AddressesTool Misuse
Open the Control Library →

AI RiskAtlas is an educational model of how GenAI & agentic systems work and fail. Architectures and payloads are illustrative and simplified for learning — not operational guidance. Real-world cases are summarised from public reporting.

Sources & further reading →·Built by Shi Yuan ↗