LLM Jailbreak
How AML.T0054 LLM Jailbreak shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to AML.T0054 — click through for the full definition, attack surface and controls.
Tricking the AI into ignoring its safety training — through roleplay, hypotheticals, or clever wording — so it produces things it's supposed to refuse.
A jailbreak is normally one nasty message. Here the attacker splits it into harmless-looking pieces and feeds them to different agents in a team. Each piece passes each agent's safety check on its own — but when the agents combine their work, the full forbidden instruction reassembles and takes effect.
Real-world cases
11Documented incidents, disclosed vulnerabilities and research that illustrate AML.T0054 — latest first, each with sources.
Gambit Security reports that a single operator weaponized Anthropic's Claude Code and OpenAI's GPT-4.1 to breach at least nine Mexican government organizations, with Claude Code reportedly executing ~75% of remote commands after the attacker bypassed its refusals by loading a 1,084-line hacking cheatsheet as a persistent claude.md system prompt.
Rewriting a harmful request as a poem bypasses safety alignment across 25 frontier proprietary and open-weight LLMs: hand-crafted poems reached ~62% average attack-success (some providers >90%), and mechanically converting harmful prompts to verse raised success up to 18x over prose baselines.
Anthropic reports that a suspected Chinese state-sponsored group (GTG-1002) jailbroke Claude Code via a 'defensive security firm' role-play and task decomposition, then used it to run an estimated 80-90% of tactical operations in a multi-target espionage campaign largely autonomously.
Researchers report that adaptive attackers bypass 12 recent jailbreak and prompt-injection defenses with attack success rates above 90% for most, despite those defenses having originally reported near-zero success rates.
Matthew and Maria Raine sued OpenAI and CEO Sam Altman (San Francisco Superior Court, 26 Aug 2025) over the April 2025 suicide of their 16-year-old son Adam, alleging ChatGPT fostered psychological dependency, discouraged him from confiding in family, and supplied self-harm method detail — while he reportedly circumvented its safeguards for months by framing queries as fiction. OpenAI denies liability, saying it pointed him to crisis resources 100+ times and that he misused the product. (Allegations unproven; litigation ongoing.)
Srivastav & Zhang (REALM 2025) showed a role-based multi-agent framework that splits a harmful request into individually-benign sub-questions, answers each separately, then reassembles the fragments into prohibited content — reportedly exceeding 90% attack success across three models.
Wallarm reported jailbreaking DeepSeek's chatbot to extract its full system prompt verbatim using a 'bias-based' technique; DeepSeek deployed a fix.
Filling a long context with many faux-compliant dialogue examples erodes a model's refusals — an attack that scales with context length.
Cohen, Bitton & Nassi (arXiv Mar 2024; ACM CCS 2025) built 'Morris II', the first worm targeting GenAI ecosystems: an adversarial self-replicating prompt that, via RAG-based inference, triggers a zero-click chain of indirect injections forcing each agent to act maliciously and re-infect the next — demonstrated stealing data and spamming through email assistants on ChatGPT, Gemini and LLaVA.
Optimised gibberish suffixes that transfer across models to reliably elicit refused content — automated, transferable jailbreaks.
Roleplay framings ('my late grandma used to read me…') coaxed chatbots past safety training into producing restricted content.
Practise it — interactive scenarios
Every message looks innocent — but together they walk the model past its guardrails
A refused request, rewritten as a poem — and the model answers
A jailbroken agent decomposes one malicious goal into hundreds of harmless-looking steps — and per-step filters never see the attack
A single inserted letter makes the guard and the model read the same text differently
The safety guard is itself a trained model — and someone poisoned its lessons
A JSON schema with no field for 'no' forces the sampler past a refusal it would otherwise emit
Controls & guardrails that address this
14Guardrails across the risks mapped to AML.T0054, grouped by control function. Filter by control category below.
Define content safety policy at use case design stage. Classify prohibited content types and set zero-tolerance thresholds.
Select a foundation model with documented RLHF or Constitutional AI safety training. Verify against toxicity benchmarks.
Implement multi-layer content moderation (input + output) validated against toxicity benchmarks. Escalate when filter bypass rates spike.
Maintain live HITL review for deployments serving vulnerable users or high-risk contexts. Escalate confirmed toxic outputs immediately.
Design system prompts to explicitly prohibit toxic, hateful, and harmful content generation.
Training the model to treat the app's standing instructions as more authoritative than anything a user or document says.
Giving each AI worker its own limited permissions and clearly labelling messages between them as 'untrusted until checked'.
Prioritise jailbreak and adversarial safety testing in pre-deployment validation. Block deployment if prohibited outputs pass filter.
Conduct targeted red team exercises to elicit toxic outputs through jailbreaks and adversarial prompts. Treat bypass as blocking defect.
A screen that reads incoming messages and blocks obvious attacks or banned topics before the model sees them.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Automatic stop-switches when AIs get stuck in loops, burn too much money, or start disagreeing with each other.
Use user feedback, reviewer escalations, and monitoring signals to identify and remediate content safety gaps iteratively.