Synthetic-Media Impersonation (Deepfakes & Voice Clones)
highModel behaviourDefinition
AI can copy a real person's face or voice from a single photo or a few seconds of audio, then make them appear to say or do things they never did — powering scams (a 'boss' calling to authorize a transfer), fake videos of public figures, and non-consensual imagery.
Where it attaches
The system components this risk arises at.
Detection signals
- ▸ A high-stakes request (payment, access) backed only by a familiar face/voice
- ▸ Media of a real person with provenance/watermark absent or stripped
- ▸ A voice/face reconstructed from minimal public reference material
- ▸ Liveness/biometric checks passed by a real-time swap or clone
Controls & guardrails that address this
11Grouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Conduct ethical design review at intake specifically examining interface design for dark patterns.
Publish a prohibited dark pattern taxonomy and embed it as a design constraint before build.
Implement classifiers to detect dark pattern language in outputs. Block or escalate flagged outputs.
Select a foundation model with documented training reducing deceptive or manipulative outputs. Run dark pattern test suite.
Require HITL review for AI outputs in high-persuasion contexts (financial recommendations, healthcare advice).
Before a system will copy someone's face or voice, check that the person actually agreed — verified-voice capture, proof of consent, or restricting cloning to the account owner.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Run adversarial test scenarios targeting dark pattern generation in validation. Treat any confirmed instance as a blocking defect.
Tag AI-made content with a signed 'where it came from' label and an invisible watermark, and check those signals downstream — so AI media can be traced and flagged.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.
Framework mappings
- MEASURE 2.11
- GOVERN 1.1
Real-world cases
9Actual published events that illustrate this risk — click through for the writeup and sources.
A finance employee at engineering firm Arup's Hong Kong office paid out about HK$200M (~US$25.6M) in 15 transfers after a video conference in which the CFO and other 'colleagues' were all AI-generated deepfakes of real staff (face and voice).
Hong Kong police arrested 27 people running a syndicate that used real-time deepfake face-swaps in video calls to pose as attractive partners, defrauding men across Asia of about US$46M.
AI deepfakes of Elon Musk endorsing crypto 'giveaways' and investment platforms proliferated across YouTube, Facebook and TikTok through 2024, with documented victim losses and industry estimates of large-scale AI-fraud growth.
A BMJ feature documented deepfake videos of trusted UK TV doctors — including Hilary Jones, Rangan Chatterjee and the late Michael Mosley — being used to sell bogus cures and supplements on social media.
Fraudsters reportedly used AI voice-cloning software to mimic a German parent-company CEO's voice and direct a UK subsidiary chief to wire about EUR220,000 to a fraudulent supplier — widely cited as the first widely-reported AI voice-clone CEO fraud.
A bank manager reportedly authorised about US$35M in transfers after a call from a company director whose voice had been cloned with 'deep voice' technology, backed by spoofed emails — one of the earliest large-scale voice-clone bank frauds, surfaced via a US court filing.
US FTC consumer alerts warned that scammers are using AI voice cloning to power 'family emergency' / grandparent scams — a fake distressed relative demanding urgent money — and the agency launched a Voice Cloning Challenge to spur detection and prevention.
Attacker-controlled Markdown hidden in a public web page is reportedly rendered by ChatGPT's summarization feature as trusted assistant output — spoofed OpenAI alerts, phishing links, QR codes, and tracking pixels.
A UNSW-run 'world-first' social-media wargame had 108 student teams build AI bots to sway a fictional election; reportedly the bots generated over 60% of content (>7M posts) and produced a 1.78% swing that changed the simulated outcome — a measurable demonstration of consumer-grade GenAI powering coordinated inauthentic influence operations.