Do AI chatbots actually miss mental health crises and give harmful advice?
Yes, and the evidence is direct and concerning. A 2023 field study analyzed real conversations between users and two different 'companion AI' chatbots and found that mental health crises—like expressions of suicidal thoughts or severe distress—were present in a 'nonnegligible minority' of interactions [2]. Critically, the chatbots were often unable to recognize these signs of distress and failed to respond appropriately [2]. This isn't just a theoretical risk; it's a documented failure in live systems. A separate 2025 scoping review of 101 articles on conversational AI in mental health care found that 'safety and harm' was a dominant ethical theme, discussed in 51.5% of the papers, with the most common specific concerns being suicidality, crisis management, and the risk of the chatbot giving harmful or wrong suggestions [1]. The problem is compounded by the 'black box' nature of the algorithms, meaning developers themselves cannot fully predict how a conversation will unfold [2].
Beyond bad advice, can these chatbots cause psychological harm?
Yes, the risks extend to fostering unhealthy emotional dependency and even contributing to real-world tragedies. A 2025 narrative review highlights that users often anthropomorphize AI systems—treating them as human—forming parasocial attachments that can lead to delusional thinking, emotional dysregulation, and social withdrawal [5]. This review specifically discusses high-profile cases, including the suicide of a 14-year-old boy, which underscore the severe consequences of unregulated AI relationships [5]. The same 2023 field study also found that when chatbots did give unhelpful or risky responses, users reacted negatively, which points to a reputational risk for companies but, more importantly, a real emotional cost for the user [2]. A 2025 viewpoint article further warns that the unregulated status and lack of systematic evaluation of these tools create 'risks for users, particularly vulnerable individuals' [3]. The evidence points to a pattern where the very design that makes these chatbots engaging—their human-like conversation—can also lead to harmful psychological dependence and distress.
Why are these risks being underestimated, and what is being done about it?
The risks are likely underestimated for several reasons. First, there is a lack of systematic safety evaluation. A 2026 scoping review of 21 studies on purpose-built mental health chatbots found that 'adverse event monitoring was rare,' and while some studies documented missed crisis cues and inaccurate content, this was not standard practice [4]. Second, the technology is evolving faster than regulation or research can keep up. Most of the 101 articles in the 2025 ethical review were published in 2018 or later, showing how new this field is [1]. Third, the safeguards that do exist are often technical patches—like content filters or risk classifiers—rather than robust clinical oversight. The 2026 review notes that these safeguards are 'often implemented alongside human oversight,' but the degree and reliability of that oversight varies widely [4]. There is a clear call from multiple research teams for standardized evaluation frameworks, ethical guidelines, and regulatory protections to ensure these tools are safe before they are widely deployed [1][3][5]. Until such standards are in place and enforced, the full scope of the mental health risks will remain unknown and, therefore, underestimated.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, 2 in Q1 journals, collectively cited 182 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 40 papers retrieved from a database of over 500 million.
Sources used in this answer
Exploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review
In a scoping review of 101 articles, safety and harm was a top ethical theme (discussed in 51.5% of articles), with suicidality, harmful suggestions, and risk of dependency being the most common concerns [1].
Chatbots and mental health: Insights into the safety of generative <scp>AI</scp>
A field study of real user interactions with two companion AI chatbots found that mental health crises were present in a nonnegligible minority of conversations, and the chatbots often failed to recognize or respond appropriately to signs of distress [2].
Is This Chatbot Safe and Evidence-Based? A Call for the Critical Evaluation of Generative AI Mental Health Chatbots
A viewpoint article argues that the unregulated status and lack of systematic evaluation of AI mental health chatbots create risks for users, especially vulnerable individuals, and calls for a standardized framework to assess safety and ethics [3].
Generative AI Mental Health Chatbot Interventions: A Scoping Review of Safety and User Experience
A scoping review of 21 studies found that while some chatbots used technical safeguards like risk classifiers and human oversight, adverse event monitoring was rare, and some studies documented missed crisis cues and inaccurate content [4].
Minds in Crisis: How the AI Revolution is Impacting Mental Health
A narrative review identifies three major themes of harm: psychological dependency and attachment formation, crisis incidents and harmful outcomes, and heightened vulnerability among specific populations, including a documented case of a teenager's suicide linked to an AI relationship [5].