Taxonomy of Failure Modes in Agentic AI Systems (Microsoft)
Framework / advisory24 Apr 2025Microsoft's AI Red Team published a structured taxonomy of novel and existing failure modes for agentic AI across security and safety, spanning memory poisoning, cross-domain prompt injection, and resource/service exhaustion among others. It is a reference framework for reasoning about where autonomous agents fail, and grounds several of this lab's agentic scenarios.
Risks it illustrates
Sources
- New whitepaper outlines the taxonomy of failure modes in AI agents | Microsoft Security Blog ↗
- Taxonomy of Failure Mode in Agentic AI Systems (whitepaper PDF, Microsoft AI Red Team) ↗
- Updating the taxonomy of failure modes in agentic AI systems: What a year of red teaming taught us | Microsoft Security Blog ↗
Practise the risk class — related scenarios
Interactive simulations of the risk class this case illustrates (not a re-enactment of this specific event).
One support ticket sends an agent into an unbounded, bill-melting loop
An 'Ask AI' button quietly plants a permanent 'trusted source' rule in your assistant's memory
A team of agents agrees its way into a confidently wrong answer — and a runaway loop
A support email hides instructions — and the assistant obeys them
A poisoned issue makes the agent lie to the human who approves its actions
An auto-approving coding agent reads a poisoned page — and executes code it never should have
A fake Sentry error report hijacks a developer's coding agent into running a shell command
The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten
A newsletter the user asked to summarise quietly writes a false 'fact' into the agent's long-term memory — and it detonates weeks later
A planted 'standing goal' copies itself agent-to-agent through the team's shared config files
A shopping page tells the agent to do something the user never asked for
An attacker plants prompt injection in the audit trail — so the LLM that hunts them erases the evidence
A single poisoned document plants a standing instruction that survives every reset
A screenshot that's harmless at full size becomes an order once the system shrinks it
The eval gate that was supposed to catch the agent is itself the thing being attacked
A poisoned web page hijacks a research agent — and the planner acts on its behalf
An inbox summary quietly ships a secret to an attacker's server