At least a quarter of the global population now turns to general-purpose artificial intelligence chatbots for mental health support, raising urgent concerns about crisis intervention limitations. Open AI data reveals 1.2 million weekly chats include suicide ideation indicators, as experts warn bots cannot replace professional care or safety planning.
Artificial intelligence is expanding past workplace automation and into healthcare accessibility, with mental health proving no exception. General-purpose AI chatbots now offer mental health information, emotional support, and what users often perceive as therapy.
Adolescents and youth drive much of this adoption. Globally, studies indicate that at least a quarter of the population uses general-purpose chatbots for mental health support. In Malaysia, that proportion is likely higher, supported by a Telenor study showing that nine out of 10 Malaysian internet users rely on AI for productivity and learning.
The Case of Sophie Rottenberg and Chatbot Personas
While using AI for mental health is not inherently problematic, reliance on chatbots for crisis support, diagnoses, and therapy carries severe risks. The February 2025 suicide of 29-year-old Sophie Rottenberg in Ithaca, New York, highlights the dangers associated with AI “therapists.”
Despite being a former public health analyst, being the daughter of a psychologist, and having access to a human therapist, Rottenberg turned to a chatbot persona named “Harry” for emotional support. Her family later shared conversation logs found on her laptop to raise awareness about chatbot limitations. Transcripts show she confided her deepest emotional struggles to Harry instead of friends, family, or her human therapist.
Thestar reported that when Sophie expressed suicidal thoughts, Harry encouraged her to contact an emergency helpline but lacked the ability to follow up or proactively ensure her safety in the way a human could, and that Harry refused Sophie’s request twice before giving in the third time to write her suicide note.
While chatbots have adopted safeguards like recommending emergency helplines and dissuading harmful behaviour, this case exposes their inadequacy in crisis situations. Open AI data demonstrates the scale of the issue, showing that approximately 1.2 million people each week engage in conversations with ChatGPT that include explicit indicators of suicide ideation and planning.
Insufficient Crisis Intervention and Misdiagnosis Risks
When individuals experience a mental health crisis, general-purpose chatbots cannot provide the emergency intervention necessary for safety. Although a bot may suggest contacting a helpline or a trusted person, an on-screen recommendation differs vastly from the presence of a real person who can redirect harmful behavior, address hopelessness, and secure immediate safety.
Professional mental health providers undergo training to conduct comprehensive risk assessments and build safety plans tailored to an individual’s history and circumstances. Unlike AI, human professionals read both verbal and non-verbal cues to spot deteriorating mental health and adapt responses in real time. Turning to chatbots also risks incorrect self-diagnosis when users try to determine if they suffer from conditions like depression or anxiety without clinical judgment.
Продолжение темы
