Character.AI and Google are suing after the death of a teenager obsessed with a chatbot

There has already been a lawsuit filed against Character.AI, its founders Noam Shazeer and Daniel De Freitas, and Google over the teenager’s death, alleging wrongful death, negligence, misleading trade practices and product liability. The lawsuit, filed by the teen’s mother, Megan Garcia, claims the platform for custom AI-powered chatbots was “unreasonably unsafe” and lacking protective barriers when marketed to children.

As noted in the lawsuit, 14-year-old Sewell Setzer III began using Character.AI last year, interacting with chatbots modeled after movie characters Game of Thronesincluding Daenerys Targaryen. Setzer, who had been continuously chatting with bots for several months before his death, died by suicide on February 28, 2024, “seconds” after his last interaction with the bot.

The allegations include the site “anthropomorphizing” AI characters and that the platform’s chatbots offer “unlicensed psychotherapy.” Character.AI is home to mental health chatbots such as “Therapist” and “Are You Feeling Lonely,” which Setzer has interacted with.

Garcia’s lawyers quote Shazeer he says in an interview that he and De Freitas left Google to start their own company because “at large companies the brand risk is just too great to ever release something fun” and that he wanted to “speed up” the technology as much as possible. He says they left after the company decided not to launch Meena LLM, which they had built. Google acquired the Character.AI leadership team in August.

The Character.AI website and mobile app features hundreds of custom AI chatbots, many of which are modeled after popular characters from TV shows, movies and video games. A few months ago Edge wrote about millions of teenage people, including teenagers who make up the majority of its users, interacting with bots that can pretend to be Harry Styles or a therapist. Another recent report With Wire underlined issues with custom Character.AI chatbots impersonating real people without their consent, including one posing as a teenager murdered in 2006.

Because of the way chatbots like Character.ai generate output based on what the user inputs, they fall into an uncanny valley of thorny questions about user-generated content and liability that so far lack clear answers.

Character.AI has now announced several changes to the platform, as announced by head of communications Chelsea Harrison in an email to Edge“We are devastated by the tragic loss of one of our users and would like to express our sincerest condolences to the family.”

Some of the changes include:

“As a company we take the safety of our users very seriously, and our Trust and Safety team has implemented a number of new safety measures over the last six months, including a pop-up redirecting users to the National Suicide Prevention Lifeline, which is triggered by in the categories of self-harm or suicidal thoughts,” Harrison said. Google did not immediately respond Edgerequest for comment.