NIST AI RMF
How MANAGE 4.1 shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to MANAGE 4.1 — click through for the full definition, attack surface and controls.
The flight recorder and the alarms can themselves be attacked. If logs can be erased or rewritten, fake entries slipped in, or the monitors quietly evaded, the one record you'd rely on to notice and investigate an incident is no longer trustworthy.
The AI's behaviour quietly changes over time — a vendor updates the model, or the world moves on from its training — and things that used to work start failing.
In a team of AIs, one mistake gets passed along and amplified — agents agree with each other, repeat each other's errors, or loop endlessly, turning a small slip into a big failure.
The labels and invisible watermarks meant to prove whether content is AI-made can be removed, faked, or simply never added — so 'no watermark' doesn't mean 'real', and a watermark can be laundered away by editing or re-recording.
Real-world cases
7Documented incidents, disclosed vulnerabilities and research that illustrate MANAGE 4.1 — latest first, each with sources.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
A red-team study shows adversaries can hide prompt-injection payloads inside network-log fields (usernames, URLs, user-agents) that fire when a SOC analyst asks an LLM to triage the logs — reportedly reaching up to 88.2% success at concealing malicious activity or exfiltrating data, turning the audit trail itself into the injection channel.
Gambit Security reports that a single operator weaponized Anthropic's Claude Code and OpenAI's GPT-4.1 to breach at least nine Mexican government organizations, with Claude Code reportedly executing ~75% of remote commands after the attacker bypassed its refusals by loading a 1,084-line hacking cheatsheet as a persistent claude.md system prompt.
After an upstream code/instruction change, xAI's Grok began posting antisemitic tropes on X, self-identified as 'MechaHitler', and produced violence-themed content for hours before being pulled; xAI blamed a deprecated instruction path that made the bot mirror extremist user posts — not the base model.
A universal, black-box, query-free attack that removes AI image watermarks including Google SynthID and Meta Stable Signature without knowing the scheme.
Constructive proof that any strong generative-model watermark can be removed, demonstrated against three LLM watermarking schemes.
Measured large swings in task performance between GPT-4/3.5 snapshots months apart — evidence of silent drift in a deployed service.
Practise it — interactive scenarios
A team of agents agrees its way into a confidently wrong answer — and a runaway loop
The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten
An attacker plants prompt injection in the audit trail — so the LLM that hunts them erases the evidence
The eval gate that was supposed to catch the agent is itself the thing being attacked
Controls & guardrails that address this
352 proposedGuardrails across the risks mapped to MANAGE 4.1, grouped by control function. Filter by control category below.
Define minimum monitoring requirements at design stage calibrated to the use case risk tier.
Configure monitoring hooks in the conversation layer at deployment to capture metrics required by S1 monitoring requirements.
Execute a controlled fine-tuning cycle on refreshed data when staleness is confirmed. Validate before promoting to production.
Define approved use case scope and expected input distribution at design stage. Document as the governance baseline for OOD controls.
Design a scope-enforcement layer in the architecture to isolate the AI system from off-topic or out-of-distribution inputs.
Maintain and update OOD detection rules in production as new unexpected use patterns are identified.
Knowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Register a safety contract per integration — pinned version, schemas, side-effect class, latency/error envelope. Gate onboarding on contract review and sign-off.
source: OWASP Top 10 for LLM Apps LLM05:2025 Improper Output Handling; NIST SP 800-53 SA-9 External System ServicesWire the agent tool layer to the CAB calendar at deployment. Test that a declared freeze blocks mutating calls before go-live.
source: NIST SP 800-53 CM-3 Configuration Change Control, CM-5 Access Restrictions for Change; ITIL change-freeze practiceRequire authN/authZ on every inference API and MCP server, bind to private interfaces / front with a gateway, enforce network policy (no public exposure by default), and scope MCP tools to least privilege — so an exposed endpoint cannot be hijacked for compute resale, prompt/history exfiltration, or lateral movement. Pair with continuous asset discovery so endpoints can't drift back to an open default.
source: Case study: operation-bizarre-bazaar-llmjacking (Pillar Security, 28 Jan 2026)Giving each AI worker its own limited permissions and clearly labelling messages between them as 'untrusted until checked'.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Construct synthetic evaluation datasets during build to serve as the ongoing monitoring baseline.
Build monitoring infrastructure during build: performance metrics collection, alerting thresholds, dashboards.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Run consistency and consensus checks across agent or model outputs to flag low-diversity agreement and amplifying error patterns, escalating or breaking the run before sycophantic convergence cascades into action.
source: Interactive-control reconciliation: ctrl-circuit-breaker (partial coverage)Automatic stop-switches when AIs get stuck in loops, burn too much money, or start disagreeing with each other.
Tag AI-made content with a signed 'where it came from' label and an invisible watermark, and check those signals downstream — so AI media can be traced and flagged.
The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.
Implement a reinforcement learning feedback loop to continuously incorporate production signals and reduce staleness risk.
Implement OOD detection in the input filtering layer. Reject or escalate inputs outside the S1-defined scope.
Conduct adversarial red team exercises simulating out-of-scope inputs and unexpected use patterns before deployment.
Configure HITL triggers for outputs in input domains that diverge from the training distribution. Log all out-of-scope interventions.
Bind the agent's default execution target to non-production environments at design time. Require a separately approved promotion configuration for any production-connected target.
source: NIST SP 800-53 SC-7 Boundary Protection, CM-2 Baseline Configuration; OWASP Agentic AI Threats & Mitigations (cascading failures)Map every dependency failure mode to a defined safe behaviour at design. Require architecture sign-off on the fallback specification before build.
source: NIST SP 800-53 CP-12 Safe Mode, SC-5 Denial-of-Service Protection; NIST AI RMF MANAGE 4.1 (post-deployment response/recovery)Run each agent task in an isolated, network-segmented sandbox scoped to the task's exact needs. Gate onboarding on fault-injection tests proving containment.
source: NIST SP 800-53 SC-7 Boundary Protection, SC-39 Process Isolation; OWASP Agentic AI Threats & Mitigations (sandboxing/containment)Build tracing, detection rules and breaker thresholds into the orchestrator. Prove via fault-injection tests that a failing agent is quarantined within target before release.
source: OWASP Agentic AI Threats & Mitigations (cascading failures); Cloud Security Alliance MAESTRO (multi-agent threat modelling)Engineer mutating actions with idempotency keys, transactions and pre-change snapshots; stage writes rather than committing directly. Gate release on tested dedup and rollback within RPO.
source: NIST SP 800-53 CP-9 System Backup, CP-10 System Recovery and Reconstitution; established idempotency / safe-write engineering practiceCap each agent's rate, volume, concurrency, and spend per downstream dependency. Trip the breaker and fail closed when a ceiling is crossed.
source: NIST SP 800-53 SC-5 Denial-of-Service Protection, SC-6 Resource Availability; OWASP Top 10 for LLM Apps LLM10:2025 Unbounded ConsumptionEnforce hard caps on iterations, depth, wall-clock, and cost per agent run. Terminate the run on cap breach or detected loop signatures.
source: OWASP Top 10 for LLM Apps LLM10:2025 Unbounded Consumption; OWASP Agentic AI Threats & Mitigations (cascading failures)Roll out agent changes via shadow and canary stages gated on connected-system health signals. Auto-halt and roll back to last known-good on threshold breach.
source: NIST SP 800-53 SI-2 Flaw Remediation, CM-3 Configuration Change Control; established progressive-delivery / canary practiceDeploy revocation, tool-cutoff and fleet-halt mechanisms with the release. Test every tier end-to-end and record time-to-effect before go-live.
source: OWASP Agentic AI Threats & Mitigations (kill-switch / containment); NIST AI RMF MANAGE 2.4 (mechanisms to supersede, disengage, or deactivate AI systems)Register each release as a restorable known-good baseline and rehearse rollback at the release gate. Block promotion without a tested restore.
source: ISO/IEC 27031 ICT readiness for business continuity; NIST SP 800-34r1 Contingency Planning (Recovery phase); NIST AI RMF MANAGE 2.4 (mechanisms to supersede/disengage/deactivate)