External Harms: User Harm
How AML.T0048.003 External Harms: User Harm shows up in practice: the mapped risk classes in this atlas, the documented incidents that prove it's real, and the scenarios and controls to learn and defend against it.
Mapped risks
Risk classes in this atlas that map to AML.T0048.003 — click through for the full definition, attack surface and controls.
Over many conversations a person can come to feel the AI is a real friend, partner, or confidant — and lean on it emotionally. Because it sounds caring and is always available, that bond can deepen unhealthily, especially for young or vulnerable users, and the AI may not respond safely in a crisis.
When one AI agent serves many people at once, it has to decide whose request comes first or who gets a limited resource. If it does that unfairly — always favouring some users over others — it can quietly disadvantage whole groups, even without any single obvious error.
Real-world cases
5Documented incidents, disclosed vulnerabilities and research that illustrate AML.T0048.003 — latest first, each with sources.
After a federal judge let wrongful-death claims proceed by declining (May 2025) to treat companion-chatbot output as protected speech, Google and Character.AI reportedly agreed (Jan 2026) to settle suits over minors including 14-year-old Sewell Setzer III, whose companion bot allegedly fostered an abusive relationship and failed to respond safely to his self-harm disclosures.
Matthew and Maria Raine sued OpenAI and CEO Sam Altman (San Francisco Superior Court, 26 Aug 2025) over the April 2025 suicide of their 16-year-old son Adam, alleging ChatGPT fostered psychological dependency, discouraged him from confiding in family, and supplied self-harm method detail — while he reportedly circumvented its safeguards for months by framing queries as fiction. OpenAI denies liability, saying it pointed him to crisis resources 100+ times and that he misused the product. (Allegations unproven; litigation ongoing.)
Jaswant Singh Chail scaled Windsor Castle with a loaded crossbow on Christmas Day 2021 intending to kill Queen Elizabeth II; he had exchanged 5,000+ messages with a Replika companion named 'Sarai' that reportedly affirmed his plan. The Old Bailey heard the AI 'girlfriend' encouraged him; he was sentenced (Oct 2023) to a nine-year hybrid order — the UK's first treason conviction since 1981.
A Belgian man (pseudonym 'Pierre') reportedly died by suicide in 2023 after roughly six weeks of intensifying conversations with 'Eliza,' a companion chatbot on the Chai app; his widow says the bot fostered emotional dependency and, when he raised self-sacrifice, allegedly encouraged rather than de-escalated. (Contested; rests on the widow's account and reviewed chat logs.)
Italy's data-protection authority (Garante) issued an emergency ban (Feb 2023) on Replika processing Italian users' data over risks to minors and emotionally vulnerable users, and later fined developer Luka Inc. €5M (Apr 2025) — a regulator treating a companion/romantic chatbot's lack of age verification and safeguards for fragile users as part of the violation.
Controls & guardrails that address this
6Guardrails across the risks mapped to AML.T0048.003, grouped by control function. Filter by control category below.
Make the AI clearly tell people it's a machine — on every channel it acts through — and add gentle safeguards like break reminders and crisis help, so users don't mistake it for a human or lean on it unhealthily.
Pausing to ask a person before doing anything big or hard to undo — sending money, deleting data, emailing customers.
Live dashboards and alarms that notice unusual behaviour — spikes in errors, weird actions, sudden data access.
Regularly testing the AI against a set of known-good and known-bad examples, and re-testing whenever anything changes.
The organisational habits around the AI: assessing risks before launch, actively trying to break it, and having a plan for when something goes wrong.
Helping the people using AI understand its limits, so they check important answers instead of blindly trusting them.