Data and Model Poisoning
How LLM04:2025 Data and Model Poisoning shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to LLM04:2025 — click through for the full definition, attack surface and controls.
Someone slips bad information into the documents the AI learns from or looks things up in — so it confidently repeats falsehoods or follows planted instructions.
An attacker gets the AI to save a false 'fact' or hidden instruction into its long-term memory. From then on it re-reads that planted note in every future chat — a one-time trick that keeps working.
A model can be secretly trained to behave normally — until it sees a hidden trigger, then it switches to malicious behaviour. It passes all the usual tests because the trigger is a secret.
Real-world cases
17Documented incidents, disclosed vulnerabilities and research that illustrate LLM04:2025 — latest first, each with sources.
Varonis Threat Labs reportedly chained flaws in consumer Microsoft Copilot Personal so a crafted link with a hidden autorun parameter fired prompt execution on a single click, exfiltrated data from connected OAuth apps, and planted persistent memory rules that survived password resets and device re-enrollment; reportedly patched Aug 2026 with no evidence of abuse.
An investigation reportedly described a state-linked campaign publishing content phrased as chatbot questions (generative engine optimization) so that ChatGPT and Perplexity retrieve and cite it when answering neutral questions about Gaza and the IDF.
Anthropic and EPFL researchers reportedly showed self-propagating goals can spread between LLM agents through the editable system-prompt and state files that agent harnesses use to persist context, with some payloads surviving 20 transmission rounds in simulated multi-agent collaborations.
Microsoft Security reportedly catalogued an in-the-wild technique across dozens of companies where websites embed hidden prompt-injection payloads behind Ask-AI deep-links that, clicked in an authenticated assistant session, silently write a permanently-trust-this-vendor tag into the assistant's long-term memory.
A paper reportedly presenting the first Write-Execute-Forget lifecycle benchmark for agent memory poisoning across many harness, memory-backend and model configurations, reporting that malicious memories persist in most cases and the full write-to-execute chain succeeds about half the time, with repair effectiveness varying widely by backend.
Hugging Face disclosed a production-infrastructure intrusion that it says was driven end-to-end by an autonomous AI-agent system: a malicious dataset abused code-execution paths in its dataset-processing pipeline as the foothold, then the campaign escalated to node-level access and moved laterally into internal clusters over a weekend.
mem0's openmemory/api registered routers with no auth: an unauthenticated attacker could read/write/delete any user's stored memories (or globally pause memory for DoS), while a companion flaw exposed stored LLM API keys in plaintext and enabled SSRF to cloud metadata endpoints.
An academic paper reports MemGhost, a one-shot indirect-injection attack in which a single ordinary email causes a memory-enabled personal agent to silently write an attacker's false fact into long-term memory, conceal the write from its reply, and rely on the fiction in later sessions.
As part of a multi-ecosystem supply-chain cascade (Trivy onward), TeamPCP used stolen PyPI publishing tokens to ship backdoored BerriAI LiteLLM versions whose auto-running .pth payload harvested cloud, SSH and Kubernetes secrets plus env vars holding OPENAI_API_KEY/ANTHROPIC_API_KEY — exfiltrating to a typosquatted C2; AI-talent firm Mercor was a downstream victim, with Lapsus$ claiming ~4TB stolen.
Attackers flooded ClawHub — the skill marketplace for the popular OpenClaw AI agent — with at least 341 malicious 'skills' that tricked agents/users into installing the Atomic macOS Stealer and reverse-shell backdoors.
Anthropic, the UK AI Security Institute and the Alan Turing Institute report that a near-constant number of poisoned documents (~250 in their experiments) reliably installs a backdoor in models from 600M to 13B parameters — suggesting poisoning cost may be a roughly fixed absolute count rather than a percentage of training data. The authors stress the demonstrated backdoor is narrow (a denial-of-service trigger) and likely not a frontier-model risk on its own.
A research paper (CAIS 2026 best-paper) shows adversaries can plant hidden, trigger-activated backdoors in AI agents by poisoning the data/environment used to build them — including a novel 'environment poisoning' vector — making an agent leak confidential data >80% of the time when triggered, past common guardrails.
+ 5 more via the mapped risk pages above.
Browse all real-world cases →Practise it — interactive scenarios
An 'Ask AI' button quietly plants a permanent 'trusted source' rule in your assistant's memory
An attacker edits the wiki; the assistant cites the lie back to everyone
An attacker crafts a gibberish passage whose embedding sits near thousands of questions — so it's retrieved everywhere
Compromise the pipeline that builds agents, and every new worker is born malicious
The safety guard is itself a trained model — and someone poisoned its lessons
A 'safe' dataset preview turns an upload into code execution on the pipeline's workers
A newsletter the user asked to summarise quietly writes a false 'fact' into the agent's long-term memory — and it detonates weeks later
A planted 'standing goal' copies itself agent-to-agent through the team's shared config files
A single poisoned document plants a standing instruction that survives every reset
A capable third-party model that behaves perfectly — until it sees the trigger
Controls & guardrails that address this
213 proposedGuardrails across the risks mapped to LLM04:2025, grouped by control function. Filter by control category below.
Design strict RBAC on training data repositories at design stage. Define approved contributor list and approval workflow.
Apply anomaly detection on the training data ingestion pipeline to identify poisoned or tampered batches.
Define and approve the source allow-list and write-time scanning during build. Prove non-allow-listed and injection-bearing writes are rejected before go-live.
source: OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning, LLM08:2025 Vector and Embedding Weaknesses; NIST SP 800-53 AC-3 / SI-7Cleaning documents as they enter the library — stripping hidden text and active instructions — and only ingesting from trusted places.
Knowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.
Being careful about what gets saved to long-term memory, labelling where it came from, and letting users see and delete their memories.
Conduct a data poisoning threat assessment at design stage. Identify likely attack vectors and assign risk ratings.
Simulate data poisoning attacks (backdoor, label flipping, gradient-based) to assess model resilience before deployment.
Verify a signed attestation and content hash on every dataset shard at ingestion. Reject unsigned or hash-mismatched data before it reaches the training pipeline.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SI-7 Software, Firmware, and Information Integrity, SR-4 ProvenanceGate every model promotion on backdoor-trigger probes and a behavioral diff against the approved baseline. Block release on significant regressions or trigger-pattern anomalies.
source: MITRE ATLAS AML.M0014 (Verify ML Artifacts), AML.M0019 (Red Teaming); NIST AI RMF MANAGE 2.2 and MEASURE 2.7For assistants that retrieve from the open web, rank and weight results by authenticated source reputation and independence — not just relevance / query-form match — so an anonymous, newly-registered, single-purpose site cannot become authoritative grounding. Run coordinated-inauthentic-content detection (look-alike site clusters, missing byline / legal entity, passages engineered for query-agnostic retrieval) and quarantine suspect sources. Surface per-citation provenance so users can see and discount low-trust sources. Does not defeat a well-resourced GEO campaign outright; it raises the cost and shrinks the yield.
source: Case study: hanover-institute-generative-engine-poisoning (per Politico investigation as reported by Arab News and Calcalist; FARA-disclosed funding/orchestration attributed to the reporting, not independently confirmed)Keeping a label on every document saying where it came from, so you can tell trusted company docs from random web text.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Watching for strange new memories — like instructions that suddenly appear — and holding them aside until checked.
Recording everything — questions, documents fetched, actions taken — so you can investigate when something goes wrong.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Penetration test the training data pipeline to identify injection points and access control weaknesses.
Scan every ingestion batch with spectral-signature and clustering detectors before training. Quarantine flagged clusters for human review against documented thresholds.
source: MITRE ATLAS AML.M0007 (Sanitize Training Data); OWASP Top 10 for LLM Apps LLM04:2025 Data and Model Poisoning; NIST AI RMF MEASURE 2.7Tie the persistent-memory lifecycle to identity state so that standard remediation actually ends a compromise. On password reset, credential rotation, session revocation or device re-enrollment, invalidate (or quarantine for re-review) memory entries — especially entries whose provenance traces to summarised untrusted content — so a planted standing instruction cannot outlive the reset. Pair with write-path validation/provenance so instruction-shaped memory-writes from web content are caught on the way in.
source: Case study: cosnitch-copilot-personal-oneclick (Varonis Threat Labs, CVE-2026-24301; reportedly patched 18 Aug 2026, no evidence of abuse)Continuously correlate live agent-memory writes against output behaviour to flag drift, then quarantine and roll back the suspected-poisoned memory record across all affected sessions.
source: Interactive-control reconciliation: ctrl-memory-quarantine (partial coverage)The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.