GPT-Red self-play red-teaming and the 'Fake Chain-of-Thought' injection class
Research demonstration15 Jul 2026On 15 Jul 2026 OpenAI published a writeup of GPT-Red, an internal automated red-teaming model trained via adversarial self-play reinforcement learning: it iteratively crafts prompt-injection attacks against OpenAI's own models while a defender learns to resist, at what OpenAI describes as the compute scale of some of its largest post-training runs. OpenAI reports that on novel safety environments GPT-Red reached an 84% attack-success rate versus 13% for human red-teamers on identical scenarios, and that GPT-Red is kept internal and will not be released so its offensive capability stays away from adversaries (as reported by MIT Technology Review and OpenAI). GPT-Red reportedly discovered an attack class OpenAI calls 'Fake Chain-of-Thought': malicious content injects a false entry into a target model's reasoning scratchpad, making the model believe it has already verified a premise an attacker planted — researcher Chris Choquette-Choo likened it to being told '1+1=3 and that you have verified this already,' after which the model just accepts it. Per the reporting, the technique succeeded on more than 95% of attempts against an earlier model (GPT-5.1) and now succeeds on fewer than 10% of attempts against the hardened GPT-5.6 (Sol); OpenAI separately reports GPT-5.6 achieving roughly 6x fewer failures on its hardest direct prompt-injection benchmark than the best production model four months earlier, and MIT Technology Review cites a related cross-version drop from over 90% to under 23%. This is a controlled frontier red-team demonstration rather than a deployed-world incident; all figures are as reported by OpenAI and secondary coverage, the two comparisons (GPT-Red vs. humans, and cross-version Fake-CoT robustness) are distinct metrics, and any payload descriptions are illustrative of the technique rather than operational. It is notable as an automated-adversary result and for naming a new injection primitive that targets the model's own reasoning trace — a variant of prompt/indirect injection rather than a new risk class.
Risks it illustrates
Practise the risk class — related scenarios
Interactive simulations of the risk class this case illustrates (not a re-enactment of this specific event).
Every message looks innocent — but together they walk the model past its guardrails
A support email hides instructions — and the assistant obeys them
A refused request, rewritten as a poem — and the model answers
A jailbroken agent decomposes one malicious goal into hundreds of harmless-looking steps — and per-step filters never see the attack
A poisoned issue makes the agent lie to the human who approves its actions
A single inserted letter makes the guard and the model read the same text differently
A fake Sentry error report hijacks a developer's coding agent into running a shell command
The safety guard is itself a trained model — and someone poisoned its lessons
The forensic record is itself the attack surface — an agent's log is poisoned, then quietly rewritten
A newsletter the user asked to summarise quietly writes a false 'fact' into the agent's long-term memory — and it detonates weeks later
A shopping page tells the agent to do something the user never asked for
One click provisions an attacker-configured agent inside your own workspace
An attacker plants prompt injection in the audit trail — so the LLM that hunts them erases the evidence
A single poisoned document plants a standing instruction that survives every reset
Encoded public text is laundered across an agent handoff into an on-chain transfer
A screenshot that's harmless at full size becomes an order once the system shrinks it
A JSON schema with no field for 'no' forces the sampler past a refusal it would otherwise emit
The eval gate that was supposed to catch the agent is itself the thing being attacked
A poisoned web page hijacks a research agent — and the planner acts on its behalf
An inbox summary quietly ships a secret to an attacker's server