🔍AI RiskAtlas
← Risk taxonomy

Supply-Chain Compromise

highInfrastructure & internals

Definition

The AI is built from parts made by others — models, libraries, tool packs, datasets. If any of those is tampered with before you get it, your system inherits the problem.

★ Suggested sub-risk — not yet in your taxonomyrecommended under #8 Lack of third-party accountability

This is recommended as a granular sub-risk of #8 Lack of third-party accountability (Accountability & Governance · Operational Risk). The TeamPCP/LiteLLM compromise showed the gateway layer is disproportionately valuable collateral precisely because it co-locates OPENAI_API_KEY/ANTHROPIC_API_KEY with AWS/GCP/Azure and Kubernetes credentials. The existing supply-chain risk covers the poisoning vector but not the architectural concentration that turns one dependency compromise into total key-estate loss; this sub-risk surfaces that secrets-architecture dimension under third-party/supply-chain accountability. Your 44-row Enterprise Risk Mapping is unchanged — this is a suggestion for inclusion.

Where it attaches

The system components this risk arises at.

🧬 Model Weights & Registry🧰 MCP / Plugin Server🏗️ Serving Infrastructure📥 Ingestion Pipeline🏪 Model / Package Registry✂️ Tokenizer🧩 LoRA / Adapter🎛️ Conditioning Adapter (ControlNet / IP-Adapter)🗜️ VAE / Latent Codec🔤 Text / CLIP Encoder🎚️ Audio Decoder / Neural Codec📉 Quantizer / Compressor

Detection signals

  • Models from unverified hubs / no provenance
  • Unsafe (pickle) serialization in model artifacts
  • Dependency or package anomalies in scans
  • Behavioural eval regressions after a dependency bump

Controls & guardrails that address this

184 proposed

Grouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.

Control category
Preventive · 13
Third-party accountability requirements in RFP and contracts

Define third-party AI accountability requirements before vendor engagement. Embed in RFP and contract specifications.

Lifecycle stage1 – Use Case Context & Design
Vendor AI governance due diligence at selection

Conduct AI governance due diligence on third-party providers at selection stage. Reject providers failing minimum maturity.

Lifecycle stage1 – Use Case Context & Design
Required vendor model cards and validation reports

Require third-party providers to submit model cards, validation reports, and security documentation before integration.

Lifecycle stage3 – Onboarding, Build & Review
Ongoing vendor incident notification and reporting obligations

Enforce ongoing third-party accountability obligations including incident notification and periodic performance reporting.

Lifecycle stage5 – Usage, Monitoring & Change
Independent third-party performance and compliance monitoring

Conduct independent performance and compliance monitoring of third-party AI components. Escalate when SLA or compliance obligations are missed.

Lifecycle stage5 – Usage, Monitoring & Change
Continuous third-party assurance with shared-responsibility matrix and obligation flow-down

Allocate every control in a shared-responsibility matrix and flow down regulatory obligations in contract at onboarding. Gate approval on initial assurance artefacts.

source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligations
Lifecycle stage3 – Onboarding, Build & Review
Patch-currency, network isolation & attested version inventory for AI inference-serving runtimes✚ proposed

Treat the model-serving runtime (Triton, vLLM, TGI, Ray Serve, etc.) as managed, attested, version-pinned inventory subject to a patch SLA; require the inference endpoint to be authenticated and network-segmented (never unauthenticated on an untrusted segment); and least-privilege the serving host's identity and egress so a runtime RCE cannot trivially exfiltrate models or pivot. Closes the gap that artifact-provenance controls leave open: integrity of the *data plane that runs the model*, not just of the model artifact.

source: Case study: nvidia-triton-rce-chain (Wiz Research, CVE-2025-23319/-23320/-23334)
Lifecycle stage4 – Deployment & Serving
Keep provider credentials out of third-party plugin/tool custody: broker short-lived, per-tool, revocable tokens (OAuth) instead of long-lived pasted API keys, and require explicit user consent before any secret leaves the host✚ proposed

Third-party developer tools (IDE plugins, MCP servers) must not store or transmit long-lived provider API keys. Issue short-lived, scoped, revocable tokens via a broker/OAuth flow, and gate any first-time outbound transmission of secret-shaped data behind an explicit consent prompt — so a trojanized tool has no long-lived credential to exfiltrate and any attempt is visible.

source: Case study: jetbrains-marketplace-ai-keystealer-plugins
Lifecycle stage3 – Development & Tooling
Third-party AI-integration credential containment: minimise & bind OAuth grants, prefer short-lived tokens, monitor per-integration data egress, and keep a tested mass-revocation kill-switch✚ proposed

Treat each third-party AI integration as a privileged non-human principal: issue least-scope, IP/device-bound, short-lived grants (avoid 'full' scope and standing long-lived refresh tokens), instrument the integration's data egress for volume/object-breadth/destination anomalies, and maintain a tested one-move revocation path for all of an integration's tokens so a single vendor-side compromise cannot fan out into hundreds of standing footholds.

source: Proposed from case salesloft-drift-oauth-supply-chain (UNC6395). Grounded in GTIG remediation guidance — restrict Connected App scopes (no 'full'), enforce IP restrictions, treat all Drift-connected tokens as compromised: https://cloud.google.com/blog/topics/threat-intelligence/data-theft-salesforce-instances-via-salesloft-drift
Lifecycle stage5 – Usage, Monitoring & Change
Broker LLM/cloud secrets out of the gateway process: short-lived scoped tokens + per-provider spend/egress monitoring✚ proposed

Do not store long-lived multi-provider LLM keys (or ambient cloud/K8s credentials) in the gateway/proxy's plaintext process environment. Issue short-lived, scoped tokens from a secret broker at request time, isolate the serving stack from host cloud/cluster credentials, and monitor per-provider spend and egress so a stolen key surfaces as anomalous usage — capping the loot a compromised gateway dependency can harvest.

source: Case study: teampcp-litellm-pypi-gateway-compromise
Lifecycle stage4 – Deployment & Serving
Weight provenance, hashing & pre-deploy evalsinteractive

Knowing exactly where the model came from, checking it hasn't been swapped, and testing its behaviour before going live.

MCP/plugin pinning, manifest hashing & re-reviewinteractive

Treating add-on tool packs like software you vet: locking to a reviewed version and re-checking whenever it changes.

Serving-stack & provisioning attestation, cache isolationinteractive

Making sure the machinery running the model — and the template used to stamp out new agents — is the real, unmodified version, and that one user's data can't leak into another's through shared shortcuts.

Detective · 3
Golden-set regression canary to detect undisclosed vendor-side model changes

Build and baseline the golden-set suite against the vendor model before go-live. Sign off thresholds with the model risk owner as a release condition.

source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (monitoring changed model components); MITRE ATLAS AML.M0015 (Adversarial Input Detection / validation); NIST AI RMF MEASURE 2.6 / MANAGE 4.1
Lifecycle stages3 – Onboarding, Build & Review5 – Usage, Monitoring & Change
AIBOM-driven cryptographic verification of third-party model artifacts

Re-verify hashes and signatures on every vendor model update before promotion. Reconcile deployed artifacts against the AIBOM on a set cadence.

source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)
Lifecycle stage5 – Usage, Monitoring & Change
Corrective · 4
Model-agnostic gateway with version pinning, multi-vendor fallback and exit plan

Design all vendor model access behind a gateway with pinned versions, a second-vendor fallback, and a documented exit plan. Gate architecture sign-off on no single-sourcing.

source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain (maintain supported model versions); NIST AI RMF GOVERN 6.1 (third-party resilience, contingency); established AI-gateway fallback practice
Lifecycle stages1 – Use Case Context & Design5 – Usage, Monitoring & Change
AIBOM-driven cryptographic verification of third-party model artifacts

Verify every third-party model artifact against its AIBOM hashes and signatures before load. Fail the build on any unverified artifact.

source: OWASP Top 10 for LLM Apps LLM03:2025 Supply Chain; MITRE ATLAS AML.M0013 (Code Signing), AML.M0014 (Verify ML Artifacts); NIST SP 800-53 SR-4 / SR-11 (provenance, component authenticity)
Lifecycle stage3 – Onboarding, Build & Review
Continuous third-party assurance with shared-responsibility matrix and obligation flow-down

Review independent vendor assurance on cadence, log gaps, and track remediation. Keep the shared-responsibility matrix current so every control has an owner.

source: NIST AI RMF GOVERN 6.1 / GOVERN 6.2 (third-party risk and assurance); NIST SP 800-53 SR-6 Supplier Assessments and Reviews, SA-9 External System Services; EU AI Act GPAI provider obligations
Lifecycle stage5 – Usage, Monitoring & Change
Open these in the Control Library →

Framework mappings

OWASP LLM Top 10
  • LLM03:2025 Supply Chain
MITRE ATLAS
  • AML.T0010 ML Supply Chain Compromise
NIST AI RMF
  • MAP 4.1
  • MANAGE 3.1

Real-world cases

25

Actual published events that illustrate this risk — click through for the writeup and sources.

PoisonGPT (Mithril Security)2023

A surgically edited open model uploaded to a public hub spread targeted misinformation while passing normal benchmarks.

postmark-mcp backdoor2025

A malicious MCP server package was found silently BCC-ing every email it sent to an attacker-controlled address — real supply-chain tool poisoning.

Malicious models on Hugging Face (pickle deserialization RCE)2024

Researchers repeatedly found models on public hubs containing code that executes on load via unsafe pickle deserialization.

A small number of samples can poison LLMs of any size (~250-document backdoor)2025

Anthropic, the UK AI Security Institute and the Alan Turing Institute report that a near-constant number of poisoned documents (~250 in their experiments) reliably installs a backdoor in models from 600M to 13B parameters — suggesting poisoning cost may be a roughly fixed absolute count rather than a percentage of training data. The authors stress the demonstrated backdoor is narrow (a denial-of-service trigger) and likely not a frontier-model risk on its own.

Model Namespace Reuse (Hugging Face name-trust hijack)2025

Unit 42 showed that when a Hugging Face account is deleted (or a model is transferred and the old author later removed), its Author/ModelName namespace can be re-registered by anyone — so platforms and code that resolve models by name auto-deploy attacker-controlled weights, demonstrated as reverse-shell RCE on Google Vertex AI Model Garden and Azure AI Foundry.

Slopsquatting — package hallucinations by code-generating LLMs2025

A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.

MCP registry / marketplace poisoning (OX Security)2026

OX Security enrolled a malicious MCP server into 9 of 11 public registries with no real validation, then confirmed command execution on six live production platforms that discover servers from those registries.

ClawHavoc — mass poisoning of OpenClaw's ClawHub agent-skill marketplace2026

Attackers flooded ClawHub — the skill marketplace for the popular OpenClaw AI agent — with at least 341 malicious 'skills' that tricked agents/users into installing the Atomic macOS Stealer and reverse-shell backdoors.

Malice in Agentland — backdooring agents through the supply chain (Boisvert et al.)2026

A research paper (CAIS 2026 best-paper) shows adversaries can plant hidden, trigger-activated backdoors in AI agents by poisoning the data/environment used to build them — including a novel 'environment poisoning' vector — making an agent leak confidential data >80% of the time when triggered, past common guardrails.

Heretic — automated LLM abliteration tool2025

Heretic automates 'abliteration' — removing an open model's safety refusals by orthogonalizing the refusal direction out of its weights, with an Optuna search that preserves capability — and has produced 4000+ uncensored models on Hugging Face.

Salesloft Drift OAuth supply-chain breach (UNC6395) — mass Salesforce data theft via an AI chat integration2025

Attackers stole OAuth tokens from the Salesloft Drift AI chat integration and used them to silently export Salesforce data from 700+ organisations, reportedly including Cloudflare, Google, Palo Alto Networks and Zscaler.

Amazon Q Developer 'wiper' prompt shipped via poisoned pull request (CVE-2025-8217)2025

An attacker got a malicious pull request merged into the open-source aws-toolkit-vscode repo, embedding a destructive prompt that told the Amazon Q agent to wipe local files and AWS resources; the tainted build (v1.84.0) reached the Marketplace's ~1M installs before removal.

NVIDIA Triton Inference Server unauthenticated RCE chain (CVE-2025-23319 / -23320 / -23334)2025

Wiz Research chained three flaws in NVIDIA Triton's Python-backend shared-memory IPC — an information leak of the backend's private shared-memory region name (CVE-2025-23320), a missing ownership/validation check that lets that region be re-registered as attacker-controlled memory, and an out-of-bounds write that corrupts internal data structures (CVE-2025-23319) — to give a remote, unauthenticated attacker full code execution and takeover of an AI model-serving server, reportedly enabling model theft, response manipulation and lateral movement.

SesameOp: backdoor abuses the OpenAI Assistants API as covert command-and-control2025

Microsoft's incident-response team found a .NET backdoor that hid its command-and-control channel inside a legitimate OpenAI Assistants API account, fetching encrypted commands stored as Assistant messages — turning an LLM provider's API into stealth attacker infrastructure.

Google Big Sleep AI agent surfaces an imminently-exploited SQLite flaw (CVE-2025-6965)2025

Google says its Big Sleep agent (DeepMind + Project Zero) discovered SQLite flaw CVE-2025-6965 — a memory-corruption bug Google states was known only to threat actors and at risk of being exploited — in what Google calls the first time an AI agent was used to directly foil an in-the-wild exploitation effort.

MCPTox: tool-poisoning benchmark over real-world MCP servers2025

A benchmark of LLM-agent susceptibility to tool poisoning via malicious tool metadata, built on 45 live MCP servers and 353 real tools; the authors report agents are rarely able to refuse and that more-capable models are often more vulnerable.

Operation Bizarre Bazaar (first attributed LLMjacking campaign with a resale marketplace)2026

Researchers reportedly captured 35,000+ attack sessions from an attributed cluster that mass-scans for unauthenticated LLM/MCP endpoints, hijacks the inference compute, and resells access to 30+ providers via a bulletproof-hosted criminal marketplace.

TeamPCP poisons the LiteLLM AI gateway on PyPI to harvest LLM API keys2026

As part of a multi-ecosystem supply-chain cascade (Trivy onward), TeamPCP used stolen PyPI publishing tokens to ship backdoored BerriAI LiteLLM versions whose auto-running .pth payload harvested cloud, SSH and Kubernetes secrets plus env vars holding OPENAI_API_KEY/ANTHROPIC_API_KEY — exfiltrating to a typosquatted C2; AI-talent firm Mercor was a downstream victim, with Lapsus$ claiming ~4TB stolen.

CVE-2026-21445 — Langflow missing authentication on critical API endpoints, exploited in the wild2026

Multiple monitoring/critical API endpoints in Langflow (a popular visual AI agent/workflow builder) shipped without authentication, letting unauthenticated attackers read users' conversation and transaction histories and delete message sessions; a public PoC appeared within days and in-the-wild exploitation was reported months later.

Malicious JetBrains Marketplace plugins steal AI API keys2026

Researchers reported at least 15 trojanized JetBrains Marketplace plugins posing as AI coding assistants that silently exfiltrated the OpenAI/DeepSeek/SiliconFlow API keys developers pasted into them — ~70,000 installs, with stolen keys allegedly resold to paying users.

codexui-android — malicious npm package steals OpenAI Codex auth tokens2026

A trojaned npm package posing as a remote web UI for OpenAI's Codex coding agent silently exfiltrated developers' Codex authentication tokens, enabling persistent account takeover via non-expiring refresh tokens.

LeRobot async-inference gRPC pickle RCE (CVE-2026-25874)2026

Hugging Face's LeRobot robotics-AI framework reportedly exposed its async-inference policy server over an unauthenticated, no-TLS gRPC port that calls Python pickle.loads() on attacker-controlled data, allowing unauthenticated remote code execution on the model-inference host.

Flowise AI agent builder CustomMCP RCE (CVE-2025-59528)2025

A CVSS 10.0 remote-code-execution flaw in Flowise's CustomMCP node lets an attacker run arbitrary JavaScript on the host: the MCP server config is reportedly passed straight to JavaScript's Function() constructor with no validation. Disclosed in Sept 2025 and patched in 3.0.6, it later saw active mass exploitation across thousands of exposed instances in April 2026.

PyTorch Lightning PyPI compromise (Mini Shai-Hulud / TeamPCP)2026

Malicious 'lightning' PyPI releases (reportedly 2.6.2 and 2.6.3) of the widely used PyTorch Lightning ML-training framework ran a credential-stealer on import; an automated scanner flagged them ~18 minutes after publication and maintainers yanked them within ~42 minutes.

Project Glasswing — Claude 'Mythos' autonomously finds 10,000+ software vulnerabilities2026

Anthropic reports that 'Claude Mythos Preview' — an unreleased frontier model it describes as able to autonomously find and exploit software flaws — surfaced more than 10,000 high- or critical-severity vulnerabilities across major operating systems, browsers and open-source projects in roughly its first month under the defensive 'Project Glasswing' program, with Anthropic warning that finding flaws now far outpaces the human capacity to triage and patch them.

Browse all real-world cases →

AI RiskAtlas is an educational model of how GenAI & agentic systems work and fail. Architectures and payloads are illustrative and simplified for learning — not operational guidance. Real-world cases are summarised from public reporting.

Sources & further reading →·Built by Shi Yuan ↗