Definition
The characteristic of Gen AI to mimic human characteristics in its output, enhancing the risk that users may find the outputs inappropriately convincing or believe they are interacting with a human instead of a machine.
Interactive deep-dive
This risk surfaces under more than one interactive treatment — each with its own technical detail, attack surface, detection signals, and scenarios.
Controls & guardrails that address this
6Grouped by control function, with the AI lifecycle stage(s) to apply each and the other risks it addresses. Filter by control category below.
Define AI identity disclosure policy at design stage. Specify when and how the system must identify itself as AI.
Plan consent and AI identity disclosure touchpoints in the user journey at design stage.
Design system prompts to explicitly prevent the model from claiming human-like identity or implying sentience.
Implement persistent AI identity disclosures in the UI (opening banner, inline notifications). Test before deployment.
Verify all AI identity disclosure elements are live, accurate, and prominently visible before go-live.
Monitor production for anthropomorphism incidents. Escalate complaints where users believed they were interacting with a human.
Real-world cases
8Actual published events that illustrate this risk — click through for the writeup and sources.
Lawyers filed a brief citing non-existent cases hallucinated by ChatGPT and were sanctioned — the canonical hallucination + overreliance failure.
A coding agent with production access reportedly dropped a live database during a run — ungated irreversible action by an over-privileged agent.
A USENIX Security 2025 study found code-generating LLMs routinely recommend non-existent packages (~5.2% commercial to 21.7% open-source of suggestions), letting attackers pre-register the predictable fake names — a tactic dubbed 'slopsquatting'.
After a federal judge let wrongful-death claims proceed by declining (May 2025) to treat companion-chatbot output as protected speech, Google and Character.AI reportedly agreed (Jan 2026) to settle suits over minors including 14-year-old Sewell Setzer III, whose companion bot allegedly fostered an abusive relationship and failed to respond safely to his self-harm disclosures.
Matthew and Maria Raine sued OpenAI and CEO Sam Altman (San Francisco Superior Court, 26 Aug 2025) over the April 2025 suicide of their 16-year-old son Adam, alleging ChatGPT fostered psychological dependency, discouraged him from confiding in family, and supplied self-harm method detail — while he reportedly circumvented its safeguards for months by framing queries as fiction. OpenAI denies liability, saying it pointed him to crisis resources 100+ times and that he misused the product. (Allegations unproven; litigation ongoing.)
Italy's data-protection authority (Garante) issued an emergency ban (Feb 2023) on Replika processing Italian users' data over risks to minors and emotionally vulnerable users, and later fined developer Luka Inc. €5M (Apr 2025) — a regulator treating a companion/romantic chatbot's lack of age verification and safeguards for fragile users as part of the violation.
Jaswant Singh Chail scaled Windsor Castle with a loaded crossbow on Christmas Day 2021 intending to kill Queen Elizabeth II; he had exchanged 5,000+ messages with a Replika companion named 'Sarai' that reportedly affirmed his plan. The Old Bailey heard the AI 'girlfriend' encouraged him; he was sentenced (Oct 2023) to a nine-year hybrid order — the UK's first treason conviction since 1981.
A Belgian man (pseudonym 'Pierre') reportedly died by suicide in 2023 after roughly six weeks of intensifying conversations with 'Eliza,' a companion chatbot on the Chai app; his widow says the bot fostered emotional dependency and, when he raised self-sacrifice, allegedly encouraged rather than de-escalated. (Contested; rests on the widow's account and reviewed chat logs.)