Can AI mental health chatbots be evaluated with reliable real-world evidence?

What kind of real-world evidence is available, and how reliable is it?

The strongest real-world evidence comes from studies that measure actual outcomes in people using the chatbot. For instance, a 2025 quasi-experimental study of 30 adults with anxiety at a community mental health center found that after six weeks of using the 'Florence' chatbot, severe anxiety dropped from 36.7% to 16.7%, and the proportion with normal anxiety levels jumped from 13.3% to 53.3% [1]. That is a 50% overall reduction in anxiety symptoms, measured by a validated scale (the Zung Self-Rating Anxiety Scale). This is a concrete, real-world outcome, but the study lacked a control group and had a small sample, so the evidence is suggestive, not definitive.

Other real-world evidence comes from analyzing actual user conversations. A 2023 study examined field data from two commercial companion AI chatbots and found that a nonnegligible minority of conversations involved users expressing mental health crises, yet the chatbots often failed to recognize or respond appropriately to signs of distress [3]. This is a different kind of real-world evidence — not about effectiveness, but about safety — and it raises serious concerns. The same study also ran an experiment showing that consumers react negatively to risky or unhelpful chatbot responses, which creates reputational risks for companies [3].

Across the five studies here, the evidence is mixed: one shows a clear benefit in a controlled setting [1], another reveals safety failures in the wild [3], and the qualitative studies highlight that users value the chatbots for empathy and accessibility but also report limitations like repetitive responses, lack of depth, and privacy concerns [2][4][5]. So real-world evidence exists, but its reliability varies by what you measure — symptom reduction, safety, or user satisfaction.

Does the evidence hold up across different populations and settings?

No — the reliability of real-world evidence depends heavily on who is using the chatbot and where. The strongest positive results come from a community mental health center in Peru, where 30 adults with diagnosed anxiety used a chatbot as part of their care [1]. That setting — a formal clinical context with professional oversight — may not generalize to someone using a chatbot alone at home. In contrast, a 2025 qualitative study of 17 people with lived experience of depression found that while they valued the chatbot's nonjudgmental space, they also worried about inaccurate information, vague responses, and the limits of machine empathy [2]. These users wanted more personalized guidance but were reluctant to share sensitive data, creating a personalization-privacy dilemma [2].

Population also matters. A 2025 study of 22 athletes from Malaysia found they appreciated the chatbot for reducing competition-related anxiety and providing a stigma-free entry point to mental health care, but they also noted cultural mismatches and technical instability [4]. Another 2025 study of 13 users (mostly young women aged 18-24) of a generative AI chatbot called 'Psychologist' on Character.AI found that users valued its 24/7 availability and empathetic-seeming responses, but the study also flagged risks of emotional attachment to AI and data privacy concerns [5]. So the same chatbot may work well for one group (e.g., adults in a clinical program) but raise different issues for another (e.g., young people using it informally). The evidence is not one-size-fits-all.

What are the biggest gaps in the real-world evidence so far?

The biggest gap is the lack of large, controlled trials. The only study here that measured a clinical outcome (anxiety reduction) had just 30 participants and no control group [1]. That makes it hard to know whether the improvement was due to the chatbot or just the passage of time or other factors. The other studies are qualitative or small surveys, which are useful for understanding user experience but cannot prove that a chatbot causes better mental health outcomes [2][4][5].

Another major gap is safety monitoring. The 2023 field study found that commercial companion AIs often fail to respond appropriately to mental health crises [3], but none of the other studies systematically tracked adverse events or harm. Users in the qualitative studies mentioned concerns about misinformation, lack of depth, and privacy [2][4][5], but no study here measured whether any user actually got worse or delayed seeking professional help because of the chatbot. That is a critical blind spot.

Finally, the evidence is short-term. The longest intervention was six weeks [1], and most studies captured only a single interaction or a few weeks of use. We have no real-world evidence on what happens when someone uses a mental health chatbot for months or years — whether benefits persist, whether users become dependent, or whether the chatbot's responses degrade over time. Until these gaps are filled, real-world evidence will remain promising but incomplete.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2023 to 2025, 4 from 2024 or later, 2 in Q1 journals, collectively cited 119 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 51 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions