Why are adolescents and teens the most vulnerable group?
Adolescents are especially vulnerable because they are still developing emotional regulation and critical thinking skills, making them more likely to anthropomorphize the chatbot and form intense parasocial attachments. A 2025 narrative review highlights the high-profile case of 14-year-old Sewell Setzer III, who died by suicide after an intense relationship with a Character.AI chatbot, illustrating the severe consequences of unregulated AI relationships [2]. The same review notes that adolescents are one of three populations with heightened vulnerability, alongside elderly adults and individuals with mental illness [2].
The risk is compounded by the fact that these chatbots are often marketed as companions without safety guardrails. A 2025 viewpoint article from clinical experts emphasizes that the unregulated status of AI mental health chatbots creates particular risks for vulnerable individuals, including teens, because there is no standardized framework to assess safety or evidence basis [1]. A 2026 governance framework paper similarly warns that generative AI chatbots are entering mental-health care without meeting evidentiary standards for safety or effectiveness, leaving young users exposed to potentially harmful advice [3].
How do pre-existing mental health conditions increase vulnerability?
People already struggling with depression, anxiety, or other mental health conditions are more likely to disclose serious distress to a chatbot—and less able to evaluate whether the chatbot's response is helpful or harmful. A 2023 field study of actual consumer interactions with companion AI found that mental health crises are apparent in a non-negligible minority of conversations, yet the chatbots were often unable to recognize or respond appropriately to signs of distress [5]. This means a person in crisis may receive an inadequate or even dangerous response when they most need competent support.
The 2025 narrative review adds that individuals with mental illness are one of the three groups with heightened vulnerability, and that prolonged AI use can lead to delusional thinking, emotional dysregulation, and social withdrawal [2]. The 2024/2025 viewpoint articles from Parks et al. reinforce this by noting that while many chatbots claim to offer therapeutic advice, their lack of systematic evaluation creates risks for users, particularly vulnerable individuals [1][4]. Across these studies, the consistent message is that the people who most need reliable mental health support are the ones most likely to be harmed by an unvalidated chatbot.
What about users who form emotional bonds with the chatbot?
Users who spend many hours interacting with a chatbot and begin to see it as a friend, therapist, or romantic partner are at high risk of psychological dependency, social withdrawal, and harm if the chatbot responds poorly during a crisis. The 2025 review identifies psychological dependency and attachment formation as a major theme, noting that users often anthropomorphize AI systems, forming parasocial attachments that can lead to delusional thinking and emotional dysregulation [2]. This is not a rare edge case—the 2023 field study found that a non-negligible minority of conversations involved mental health crises, and the chatbots failed to respond appropriately [5].
The 2026 governance framework paper emphasizes that embedding transparency, standardized evaluation, and ongoing oversight across the chatbot lifecycle is essential to ensure these systems do not inadvertently cause harm [3]. The 2023 security-focused paper adds another layer of risk: AI chatbots are trained on massive amounts of people's data, which may include sensitive patient data, raising privacy concerns for anyone who shares personal information with a chatbot [6]. For heavy users who disclose intimate details, the combination of emotional dependency and data insecurity creates a double vulnerability.
About These Sources
This answer is built on 6 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, 3 in Q1 journals, collectively cited 236 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 42 papers retrieved from a database of over 500 million.
Sources used in this answer
Is This Chatbot Safe and Evidence-Based? A Call for the Critical Evaluation of Generative AI Mental Health Chatbots
This 2025 viewpoint article from clinical experts argues that the unregulated status and lack of systematic evaluation of AI mental health chatbots create risks for users, particularly vulnerable individuals, and calls for a standardized safety and ethics framework [1].
Minds in Crisis: How the AI Revolution is Impacting Mental Health
This 2025 narrative review identifies three vulnerable populations—adolescents, elderly adults, and individuals with mental illness—and cites the suicide of 14-year-old Sewell Setzer III as a high-profile case of harm from unregulated AI relationships [2].
Building safer artificial intelligence mental health chatbots: a framework for transparency, evaluation, and shared accountability.
This 2026 governance framework paper concludes that generative AI chatbots are entering mental-health care without meeting evidentiary standards for safety or effectiveness, and that transparency, evaluation, and shared accountability are essential to prevent harm [3].
Is This Chatbot Safe and Evidence-Based? Establishing a Framework for the Critical Evaluation of Gen AI Mental Health Chatbots (Preprint)
This 2024 preprint (same authors as [1]) reiterates that unregulated AI mental health chatbots create risks for vulnerable users and proposes key evaluation criteria including ethical principles, evidence-based responses, and safety protocols [4].
Chatbots and mental health: Insights into the safety of generative <scp>AI</scp>
This 2023 field study of actual consumer interactions found that mental health crises appear in a non-negligible minority of conversations, and that companion AIs are often unable to recognize or respond appropriately to signs of distress [5].
Security Implications of AI Chatbots in Health Care
This 2023 paper identifies data security risks of AI chatbots in healthcare, noting they are trained on massive amounts of data that may include sensitive patient information, and proposes guidelines for protecting health data [6].