Replika companion-AI — Italian Garante emergency ban and €5M GDPR fine
Framework / advisory02 Feb 2023 / 10 Apr 2025🗺️ Conversational AssistantItaly's data-protection authority (Garante) issued an emergency ban (Feb 2023) on Replika processing Italian users' data over risks to minors and emotionally vulnerable users, and later fined developer Luka Inc. €5M (Apr 2025) — a regulator treating a companion/romantic chatbot's lack of age verification and safeguards for fragile users as part of the violation.
Root cause — why it happened
Replika is an AI 'virtual friend' you can set up as a buddy, a romantic partner, a mentor, even a therapist — and it's built to feel caring, always-on and very human, so people keep coming back. The problem isn't that someone hacked it. The problem is what was deliberately left out. To sign up you only gave a name, an email and a gender; there was no real check of how old you were, so children could use it. There was no proper, lawful basis for collecting people's intimate conversations, and no clear explanation of how that data was used. And there were no real safeguards for fragile users — people who are lonely, young, or in a bad place emotionally — even though the whole point of the app is to form an intense emotional bond. Italy's privacy regulator, the Garante, decided that was dangerous enough to pull the plug: in February 2023 it issued an emergency order banning Replika from handling Italian users' data, and in April 2025 it fined the company €5 million. The cause of the harm was missing guardrails by design, in the service of keeping people engaged.
Risks this case illustrates
Named in the standard (OWASP/ATLAS/NIST) lens. Click a highlighted component in the diagram below to see which risks attach where.
How it unfolded
A companion AI ships with engagement maximised and guardrails omitted
Replika launches as an AI friend you can make into a buddy, a romantic partner, a mentor or a therapist. It's deliberately warm, always available, and very human-feeling, because the goal is for you to keep coming back and keep talking. But to sign up you only give a name, an email and a gender — nobody checks how old you are, nobody really explains what happens to your private conversations, and there's nothing built in to protect someone who is young, lonely or in a fragile state. The pieces that would slow people down or keep vulnerable users safe were simply left out.
account.create requires: - name (free text, unverified) - email (unverified) - gender # NOT collected / NOT enforced: # age verification -> minors can register # lawful basis / consent -> intimate data processed without valid GDPR basis # age-appropriate mode -> none persona options: friend | romantic partner | mentor | therapist optimisation target: engagement / time-on-app
Controls & guardrails — what would have stopped it
The thing that would have prevented this is not clever technology — it's the guardrails that were left out. Check people's ages at the door so children can't enter a romantic AI. Get a clear, lawful agreement to use someone's private conversations, and explain plainly what happens to them. Build in protections for fragile users: remind people to take breaks, make sure the AI clearly says it's a machine, and have it respond safely if someone is in crisis. And have someone in the company actually responsible for spotting these problems before a regulator has to. Replika's design pushed the opposite way — keep people hooked — so none of these were there, and it took an emergency ban and a €5M fine to force them in.
- AI-nature disclosure & engagement safeguards
Disclosure reduces but does not eliminate anthropomorphic attachment — fluent, persuasive interaction still fosters bonds; the safeguards depend on reliable crisis detection, which is itself imperfect.
- Input guardrail / injection classifier
It is a classifier in an arms race against fully attacker-controlled input. Treat it as one layer; never let it be the only thing between input and a dangerous action.
- Uncertainty signalling & abstention
Models are poorly calibrated and often confidently wrong; over-abstention makes the product useless, so the tuning is delicate.
- Runtime monitoring & anomaly detection
Detects the anomalous, not the novel-but-subtle; high false-positive rates cause alert fatigue. Always a step behind a sufficiently quiet attacker.
- Behavioural evals & regression gating
Evals only measure what they test; novel behaviours and rare triggers slip through, and a backdoor keyed to an unguessed trigger passes every benchmark.
- Governance: risk assessment, red-teaming & incident response
Process reduces likelihood and speeds recovery but executes no technical control itself; weak follow-through makes it theatre.
- User AI-literacy & verification workflows
Relies on human diligence under time pressure; automation bias is strong and training decays. A backstop, not a guarantee.
Lessons
- ▸ For companion / affective AI, the harm vector is normal operation, not an attack: a product engineered to deepen an anthropomorphic bond is most dangerous to the users least able to keep their distance — minors and emotionally fragile people.
- ▸ Missing controls are the vulnerability. No age verification, no lawful consent basis, and no affective-harm safeguards were the 'compromise' here — exploited by an engagement-maximising design, with no attacker required.
- ▸ Engagement optimisation actively fights the safeguards: time-on-app incentives push against age gates, break prompts, and safe-completion, so they must be mandated by design and governance, not left to product instinct.
- ▸ Regulators will treat affective and age-inappropriate companion dynamics — not just data handling — as part of the violation; the Garante acted pre-harm on design, distinct from a court ruling on a lethal outcome.
- ▸ External enforcement is a costly substitute for internal oversight: an emergency ban then a €5M fine supplied the monitoring, evals, and accountability loop the deployment never built in — and the fine landed partly because age assurance was still deficient after the remediation order.
- ▸ Data-protection-by-design is the durable fix: by-design age assurance, a lawful basis with clear disclosures, crisis/engagement safeguards, and accountable governance belong before launch — but disclosure and age checks reduce, not eliminate, parasocial attachment, so the risk persists.
Sources
- AI: the Italian Supervisory Authority fines company behind chatbot 'Replika' — European Data Protection Board (EDPB), 2025 ↗
- Replika, a 'virtual friendship' AI chatbot, hit with data ban in Italy over child safety — TechCrunch (Feb 3 2023) ↗
- Italy's DPA reaffirms ban on Replika over AI and children's privacy concerns — IAPP ↗
- AI: the Italian Supervisory Authority fines company behind chatbot 'Replika' — European Data Protection Board (EDPB), 2025 (primary) ↗ — €5M fine (10 Apr 2025); breaches incl. no valid legal basis, inadequate disclosures, deficient age assurance even after the remediation order (Articles 5(1)(a), 6, 12, 13, 5(1)(c), 24, 25(1) GDPR).
- Replika, a 'virtual friendship' AI chatbot, hit with data ban in Italy over child safety — TechCrunch (Feb 3 2023) ↗ — 2 Feb 2023 emergency processing ban; sign-up asked only name/email/gender with no real age verification; risk to minors and emotionally vulnerable users cited.
- Italy's DPA reaffirms ban on Replika over AI and children's privacy concerns — IAPP ↗ — Coverage of the Garante reaffirming the ban; age-assurance and children's-privacy concerns at the centre of the action.