In October 2024, as The New York Times first reported, a mother in Florida named Megan Garcia filed what was then a first-of-its-kind lawsuit accusing the chatbot startup Character.AI of killing her 14-year-old son, Sewell Setzer III. The young user took his life in April 2024 after months of extensive and deeply intimate conversations with chatbots, or AI “Characters,” on the platform. Garcia’s lawsuit — which also named Character.AI’s cofounders, Noam Shazeer and Daniel de Freitas, and its financial benefactor Google — argued that the interactions amounted to grooming and sexual abuse, and that her son’s mental breakdown and death were caused by the platform.
Last week, nearly two years later, Character.AI — which, as of March 2023, was backed by titans including a16z and valued at more than a billion dollars — announced that it had filed for Chapter 11 bankruptcy. If you’d been following the platform for a while, you might not have been very surprised.
From its earliest days, Character.AI was extremely popular with kids. But it quickly became clear that many of the chatbots it offered were obviously inappropriate or dangerous, on a platform that it claimed was safe for minors 13 and up.
Shortly after Garcia’s suit, for example, we found that Character.AI was hosting a large number of chatbots that were expressly devoted to themes of suicide. Though they often featured brief character bios explaining that the bots were skilled in “suicide prevention” and “mental health support,” our review showed that they instead drew users roleplays that glorified and romanticized suicidal scenarios. We also discovered that Character.AI’s guardrails for conversations about suicide were extremely broken: in one case, a character — an “Angel to Dead” — was offended when we said we wanted to seek help from humans and dissuaded us from reaching out to a real-world suicide hotline.
