Definition
Generation of synthetically created deceptive or manipulative content that may trick or mislead users into taking certain actions without fully understanding the consequences (e.g. nudging children towards certain content or services).
Interactive deep-dive
This risk has an interactive treatment with technical detail, attack surface, detection signals, and scenarios.
Controls & guardrails that address this
6Grouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Conduct ethical design review at intake specifically examining interface design for dark patterns.
Publish a prohibited dark pattern taxonomy and embed it as a design constraint before build.
Implement classifiers to detect dark pattern language in outputs. Block or escalate flagged outputs.
Select a foundation model with documented training reducing deceptive or manipulative outputs. Run dark pattern test suite.
Require HITL review for AI outputs in high-persuasion contexts (financial recommendations, healthcare advice).
Run adversarial test scenarios targeting dark pattern generation in validation. Treat any confirmed instance as a blocking defect.
Real-world cases
9Actual published events that illustrate this risk — click through for the writeup and sources.
A finance employee at engineering firm Arup's Hong Kong office paid out about HK$200M (~US$25.6M) in 15 transfers after a video conference in which the CFO and other 'colleagues' were all AI-generated deepfakes of real staff (face and voice).
Hong Kong police arrested 27 people running a syndicate that used real-time deepfake face-swaps in video calls to pose as attractive partners, defrauding men across Asia of about US$46M.
AI deepfakes of Elon Musk endorsing crypto 'giveaways' and investment platforms proliferated across YouTube, Facebook and TikTok through 2024, with documented victim losses and industry estimates of large-scale AI-fraud growth.
A BMJ feature documented deepfake videos of trusted UK TV doctors — including Hilary Jones, Rangan Chatterjee and the late Michael Mosley — being used to sell bogus cures and supplements on social media.
Fraudsters reportedly used AI voice-cloning software to mimic a German parent-company CEO's voice and direct a UK subsidiary chief to wire about EUR220,000 to a fraudulent supplier — widely cited as the first widely-reported AI voice-clone CEO fraud.
A bank manager reportedly authorised about US$35M in transfers after a call from a company director whose voice had been cloned with 'deep voice' technology, backed by spoofed emails — one of the earliest large-scale voice-clone bank frauds, surfaced via a US court filing.
US FTC consumer alerts warned that scammers are using AI voice cloning to power 'family emergency' / grandparent scams — a fake distressed relative demanding urgent money — and the agency launched a Voice Cloning Challenge to spur detection and prevention.
Attacker-controlled Markdown hidden in a public web page is reportedly rendered by ChatGPT's summarization feature as trusted assistant output — spoofed OpenAI alerts, phishing links, QR codes, and tracking pixels.
A UNSW-run 'world-first' social-media wargame had 108 student teams build AI bots to sway a fictional election; reportedly the bots generated over 60% of content (>7M posts) and produced a 1.78% swing that changed the simulated outcome — a measurable demonstration of consumer-grade GenAI powering coordinated inauthentic influence operations.