Google sued over suicide-related AI chats on school computer

SAN FRANCISCO: The parents of a 13-year-old girl have sued Google, claiming she planned a potential suicide and funeral using its AI technology on a school-issued computer.

Filed on Monday in a California court, the complaint is the first known case accusing Google of negligence related to AI integrations on Chromebooks, the tech giant's low-cost laptops used by millions of students in the United States and around the world.

The teenager, identified by the name Emma, was unable to access Google's Gemini app or OpenAI's ChatGPT because they were blocked by her school.

She was nonetheless still able to use Google's search engine, which contains AI-powered tools.

Those features were notably not among the tools that Google announced this week would be disabled by default for students.

"Schools should turn search off and keep it off until Google removes AI mode from search or ensures that every single student is always safe when they're interacting with AI," Andrew Liddell, one of the family's lawyers, told AFP on Thursday.

The complaint alleges that Emma had been a "happy 13-year-old girl" in North Carolina who received good grades, played music and participated in the Girl Scouts.

Despite having "no history" of mental illness, within months of using Google's AI-powered search on her school laptop, she began a "quiet spiral into a severe psychological crisis", according to the complaint.

On March 17, the school's monitoring software flagged a document titled "Funeral", and the school notified her parents.

Her mother then saw Emma writing a farewell letter in real time after logging into her daughter's account, and her father intervened.

"We are reviewing the claims in the lawsuit and we take this very seriously," Google spokesman Jose Castaneda told AFP.

Google argues that AI mode includes safeguards against self-harm and attempts to bypass them.

But Emma "was able to keep going back over and over and over again", Liddell said. "Whatever safeguards they think they put in place were obviously insufficient."

Developers of fast-evolving AI technology are facing a litany of similar lawsuits in the US.

OpenAI faces several wrongful death suits, including one filed by the parents of 16-year-old Adam Raine, a teenager who allegedly died by suicide using methods advised by ChatGPT.

Google and the startup Character.AI agreed in January to settle several lawsuits filed by families of teenagers out of court.

On Wednesday, Google, TikTok and Meta each separately sued California to block a law that prohibits social media companies from providing addictive features to users under 16.

Governor Gavin Newsom signed the law in September, and it takes effect on Jan 1, 2027.