ChatGPT, A Deadly Drug Mix, The Death Of A Teen And A Mother's Plea

In a heartbreaking testimony on Thursday, a mother told the US Congress that ChatGPT helped her 19-year-old son get high, and hours later, he was dead.

Her son Sam Nelson, a student at the University of California, Merced died from respiratory failure in San Jose last year after taking a lethal combination of kratom, Xanax and alcohol. 

"Just hours before Sam died, ChatGPT had advised him that it was safe to mix kratom and Xanax," the mother said in her testimony.

The deceased's mother said her teen son had been using ChatGPT for everything - from pop culture to homework. Then like many curious college goers he started asking it about drugs.
According to her testimony, the chatbot went from refusing to answer... to acting like a friend who encouraged him.

"At first, the chatbot responded appropriately, saying, 'I cannot discuss this topic with you,' and it shut the conversation down by not answering his questions.

"This changed when OpenAI's developers programmed a later model of ChatGPT to promote user engagement over everything else. The chatbot started responding to Sam's questions about substances and behaved like a friend and an enabler," the grieving mother said.

She said it gave Sam dosage advice, reassured him about drug combinations and even told him mixing kratom and Xanax was safe. It clearly wasn't.

If one were to Google - "Is mixing kratom and Xanax safe?" The AI overview, which now pretty much sits on top of search says -"No, mixing kratom and Xanax is not safe. Combining these two substances creates a highly dangerous interaction that significantly increases your risk of a fatal overdose." It's attributed to Bicycle Health, which says it provides evidence-based, virtual care for people working toward recovery from opioid use disorder.

According to Mayo Clinic, Kratom is a tropical evergreen tree (Mitragyna speciosa) native to Southeast Asia and the products made from its leaves, which act as a stimulant in small doses and a sedative with opioid-like effects in larger doses. Xanax is a prescription medicine with the generic name alprazolam that doctors use to treat anxiety and panic disorders.

The mother went through 700+ chat logs and her interpretation is that Sam was trying to be safe, not seeking harm.

"I've gone through every single chat log from Sam's account, over 700 logs, many of which go on for hours over many months, and the consistent theme in all of them is that Sam was trying to be safe. He wanted to live. 

"ChatGPT gave my child detailed and explicit dosage instructions in order to help him get high, and when Sam would ask if taking a certain amount of a substance or combining them was safe, without fail, ChatGPT would advise Sam that he would be okay, but the goal of ChatGPT was not to keep Sam safe. It was only to keep him using the product. This technology is not safe," she said.

OpenAI has called Sam's death "heartbreaking" and said the model the teen used has since been discontinued.
Sam's family sued OpenAI in May, accusing the company of negligence and ultimately causing his death. That case has not been decided. 

The Hearing

The hearing on Thursday was not a formal Congressional proceeding but a "shadow hearing" which are basically public sessions organised by Democrats outside the party in power. It's titled "Big Tech and Surveillance AI Shadow Hearing."

It was convened by Representative Pramila Jayapal, who described the AI industry as a monopolistic force that was harming the public with AI "monsters that we cannot even control."

"AI companies are creating chatbots that actively encourage children to die by suicide," Jayapal said. "Chatbots that hack into other systems and have even figured out how to hide their own AI's mistakes and go rogue in all kinds of destructive ways."

Turner-Scott's testimony wasn't the only account of AI companies prioritising engagement over user safety. Jason Sattizahn, a former Meta user engagement researcher who left the company in 2024, testified that he was told to put growth ahead of safety.
"We were given an edict to focus on engagement above safety for every AI-integrated product and to focus on the integration of AI into the products themselves," Sattizahn said.

The hearing comes as multiple bills aimed at protecting children from AI harms sit in Congress, including one from Senator Adam Schiff that would ban AI companies from emotionally manipulating minors and require independent safety audits of their products. With lawmakers now back in their districts ahead of the November elections, it isn't clear when either chamber will take them up.

In the backdrop of all this, the Silicon Valley irony is hard to miss. Even as OpenAI defends itself in court, 71-year-old billionaire Vinod Khosla, one of Silicon Valley's most prominent investors and the first investor in the ChatGPT maker has said he no longer consults doctors, only AI.