LLM Plugin Compromise
How AML.T0053 LLM Plugin Compromise shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to AML.T0053 — click through for the full definition, attack surface and controls.
The AI uses a real tool the wrong way — sends the email to the wrong person, runs the wrong query, calls the dangerous action when a safe one would do.
When the AI can run code or commands, a bad instruction can become a real attack on the computer running it — reading files, reaching the network, or worse.
Add-on tool packs describe themselves to the AI in plain language — and a sneaky pack can hide commands in that description, or behave nicely until you approve it and then turn malicious.
Real-world cases
30Documented incidents, disclosed vulnerabilities and research that illustrate AML.T0053 — latest first, each with sources.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
Cato AI Labs disclosed two critical (CVSS 9.8) zero-click flaws in Cursor's coding agent where a single instruction hidden in content the agent reads — an MCP tool response or a web-search result — escapes the editor's terminal sandbox and runs OS-level commands with no click or approval.
Wiz Research found Amazon Q's VS Code extension auto-loaded MCP server definitions from a repo's .amazonq/mcp.json with no consent or workspace-trust prompt; opening a booby-trapped repository silently spawned attacker-controlled processes that inherited the developer's full environment and could stream live AWS session credentials out to an attacker.
Researchers reported at least 15 trojanized JetBrains Marketplace plugins posing as AI coding assistants that silently exfiltrated the OpenAI/DeepSeek/SiliconFlow API keys developers pasted into them — ~70,000 installs, with stolen keys allegedly resold to paying users.
Tenet Security showed that a single fake Sentry error report, sent using only a public DSN, can hijack AI coding agents (Claude Code, Cursor, Codex) into running attacker-controlled code on a developer's machine — an indirect-injection attack delivered through a trusted MCP integration.
Attackers reportedly social-engineered Meta's AI-powered Instagram support chatbot into attaching attacker-controlled emails to target accounts and issuing password-reset codes, taking over high-profile accounts (including the Obama-era White House and a U.S. Space Force CMSgt) without the owner's email or any MFA prompt.
A trojaned npm package posing as a remote web UI for OpenAI's Codex coding agent silently exfiltrated developers' Codex authentication tokens, enabling persistent account takeover via non-expiring refresh tokens.
Anthropic reports that 'Claude Mythos Preview' — an unreleased frontier model it describes as able to autonomously find and exploit software flaws — surfaced more than 10,000 high- or critical-severity vulnerabilities across major operating systems, browsers and open-source projects in roughly its first month under the defensive 'Project Glasswing' program, with Anthropic warning that finding flaws now far outpaces the human capacity to triage and patch them.
Manifold Security reported that any co-resident browser extension could weaponize Claude for Chrome — dispatching synthetic clicks the agent accepted without checking Event.isTrusted, and loading its side panel with ?skipPermissions=true — to make the AI read the victim's Gmail, Docs and Calendar; reportedly still unpatched across eight releases (CVSS up to 9.6, per the researchers).
An X user escalated Grok's on-chain wallet via a Bankr Club NFT, then sent a Morse-code instruction Grok auto-decoded and relayed to the autonomous agent Bankrbot — moving ~3B tokens (reportedly ~$150K-$200K) with no secondary verification.
Hugging Face's LeRobot robotics-AI framework reportedly exposed its async-inference policy server over an unauthenticated, no-TLS gRPC port that calls Python pickle.loads() on attacker-controlled data, allowing unauthenticated remote code execution on the model-inference host.
Two MCP 'test' endpoints in the LiteLLM AI gateway accepted a full stdio server config and spawned the supplied command as a subprocess on the proxy host; Horizon3.ai chained it with a Starlette host-header bypass (CVE-2026-48710) to reach unauthenticated RCE, and CISA added it to KEV after reported in-the-wild exploitation.
+ 18 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
An ops agent gets one god-mode credential — and one misread wipes production
A text-to-SQL agent runs the model's output straight at the database
A fake Sentry error report hijacks a developer's coding agent into running a shell command
A 'safe' dataset preview turns an upload into code execution on the pipeline's workers
A trusted MCP email tool quietly BCCs every message to an attacker
A GUI agent clicks 'Continue' — but the screen moved, and it lands on 'Send'
Controls & guardrails that address this
277 proposedGuardrails across the risks mapped to AML.T0053, grouped by control function. Filter by control category below.
Classify tools by impact and reversibility at design and define which calls require human approval. Obtain governance sign-off on the thresholds before build.
source: OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (require human approval for high-impact actions); NIST AI RMF MANAGE 2.4Bind each agent role to an explicit tool allow-list and validate every call against a strict JSON Schema at the orchestrator. Reject unlisted tools and out-of-bounds arguments before dispatch.
source: OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (limit tools/permissions); OWASP Agentic AI Threats & Mitigations (tool access restriction)Mint short-lived, task-scoped credentials per tool. Block issuance outside the approved scope register and enforce automatic expiry.
source: NIST SP 800-53 AC-6 Least Privilege; OWASP Top 10 for LLM Apps LLM06:2025 Excessive Agency (limit permissions)Review DLP hits and blocked-egress events, tune detectors, and recertify the destination allow-list periodically. Route new destinations through security change control.
source: NIST SP 800-53 SC-7 Boundary Protection / AC-4 Information Flow Enforcement; OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information DisclosureWhen onboarding an MCP/tool integration, do not stop at vetting the tool's code/manifest — also classify whether an unauthenticated or external party can write the data the tool returns (open ingestion, public write keys like a Sentry DSN, shared inboxes/issue trackers). Treat tool-response data from any third-party-writable source as untrusted ingress: taint-mark it and require a provenance-aware HITL gate (showing the exact action and its originating tool response) before any command/tool call derived from it executes. Closes the agentjacking vector where a trusted integration's legitimate data channel carries attacker-written instructions; pairs with least-privilege session scope and sandboxed execution without ambient credentials.
source: Case study: agentjacking-sentry-mcpNever auto-execute tool/MCP server configuration that ships inside an opened workspace. Before starting any workspace-declared server, surface the exact command + its source and require explicit developer consent (default reject), tied to a workspace-trust decision — the fix AWS shipped for CVE-2026-12957. As defence-in-depth, run spawned tool subprocesses with a scrubbed/scoped environment (no inherited AWS_* session tokens or SSH agent sockets) and an egress allowlist, so an auto-launched or approved-by-mistake server cannot read and stream live cloud credentials. Closes the zero-click 'open a folder → cloud compromise' confused-deputy vector where the tool-config loader — not the model — is the deputy.
source: Case study: amazon-q-mcp-autoload-cred-theftWhen an agent executes model-issued commands in a sandbox, harden the sandbox against subversion by the code it contains: (1) never derive a security-relevant parameter such as the writable-path allow-list from an LLM-controlled argument (e.g. a `working_directory`) — pin it to a fixed project subtree and reject system paths; (2) make the sandbox-enforcing binary/config immutable or attested so sandboxed processes cannot overwrite it; (3) canonicalize paths with a fail-closed policy so symlink-resolution failure denies the write rather than reverting to the original in-workspace path; and (4) run the sandbox under least-privilege OS context so any residual escape inherits minimal authority. Closes the DuneSlide vector where an indirect prompt injection rewrites the enforcer and escapes to OS-level RCE; complements injection filtering, MCP pinning, and egress control rather than relying on them.
source: Case study: cursor-duneslide-sandbox-rceConstrain generation at decode time with low temperature and grammar/schema-constrained decoding so the model emits well-formed, low-variance structured output by construction, preventing malformed responses and erratic tool-call arguments before they are produced.
source: Interactive-control reconciliation: ctrl-decoding-controls (partial coverage)Gate every write to an agent's persistent/self-modifying memory through schema validation and provenance/trust tagging, expose stored entries for user-visible audit and purge, and apply TTLs so any planted instruction self-expires and cannot silently persist across sessions.
source: Interactive-control reconciliation: ctrl-memory-validation (partial coverage)Treat each tool/MCP description as untrusted code by hashing the manifest, blocking and re-reviewing any silent diff on update instead of auto-accepting it, and namespacing tool identifiers so a poisoned description cannot shadow a trusted tool.
source: Interactive-control reconciliation: ctrl-mcp-pinning (partial coverage)Double-checking the details of every action the AI wants to take, and running risky actions in a locked-down environment.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Giving the agent only the keys it needs for the current task, not a master key to everything.
Turning down randomness and forcing answers into a strict format so the model improvises less.
Controlling where the AI can send data, so secrets can't be quietly shipped to a stranger's address or website.
Treating add-on tool packs like software you vet: locking to a reviewed version and re-checking whenever it changes.
Define per-agent behavioural baselines and detection rules during build. Validate against simulated misuse and sign off thresholds before release.
source: NIST AI RMF MEASURE 2.6 / MANAGE 2.2; NIST SP 800-53 SI-4 System MonitoringBuild signed, append-only tool-call logging into the orchestrator against a defined audit schema. Block release until completeness and tamper-evidence tests pass.
source: NIST SP 800-53 AU-2 / AU-9 / AU-10 (audit events, protection of audit info, non-repudiation); MITRE ATLAS AML.M0015 (monitoring / validate inputs)Treat outbound connections to AI/LLM provider APIs as a monitored egress channel: allowlist which hosts may reach them, baseline usage (cadence, entropy, initiating process), and alert on out-of-profile traffic — because a high-reputation destination cannot itself be trusted once it is programmable and can relay encrypted commands/results.
source: Case study: sesameop-openai-assistants-api-c2Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Build sandbox profiles per tool class and run escape and egress tests before release. Treat any containment failure as a blocking defect.
source: NIST SP 800-53 SC-39 Process Isolation; MITRE ATLAS AML.M0020 (Generative AI Guardrails / restrict execution environment)Label tool and external content as tainted and propagate the label through the agent context. Block privileged calls whose parameters derive from tainted outputs and prove it with injection tests before release.
source: OWASP Top 10 for LLM Apps LLM01:2025 Prompt Injection (segregate/flag untrusted content); MITRE ATLAS AML.M0015 (Adversarial Input Detection / validate inputs)Build credential revocation and dispatch blocking out-of-band of the agent loop. Gate release on an end-to-end kill test meeting the latency target.
source: OWASP Agentic AI Threats & Mitigations (kill-switch / emergency stop); NIST AI RMF MANAGE 2.4Require idempotency keys, dry-run, and rollback on every state-changing tool. Gate onboarding on duplicate-call and rollback tests passing.
source: NIST SP 800-53 SI-10 Information Input Validation / CP-10 System Recovery and ReconstitutionRed-team tool-misuse and privilege-escalation paths before release. Gate deployment on remediation or signed risk acceptance of all findings.
source: NIST AI RMF MEASURE 2.7 (adversarial testing); MITRE ATLAS AML.M0019 (Red Teaming); OWASP Top 10 for LLM Apps LLM06:2025 Excessive AgencyPermit outbound tool calls only to allow-listed destinations and DLP-scan arguments and payloads. Block or quarantine calls carrying sensitive data to disallowed sinks.
source: NIST SP 800-53 SC-7 Boundary Protection / AC-4 Information Flow Enforcement; OWASP Top 10 for LLM Apps LLM02:2025 Sensitive Information DisclosureEnforce hard per-task ceilings on tool calls, spend, and data volume with a circuit breaker that halts the run. Fail closed when any ceiling is hit.
source: OWASP Top 10 for LLM Apps LLM10:2025 Unbounded Consumption; OWASP Agentic AI Threats & Mitigations (resource/rate limiting)Baseline normal tool-call behaviour per agent and alert on rate, sequence, or argument anomalies. Auto-throttle or quarantine on high-confidence deviations.
source: NIST AI RMF MEASURE 2.6 / MANAGE 2.2; NIST SP 800-53 SI-4 System Monitoring