Your feelings become data: the privacy risk of AI companions
When an AI companion is always with you, it's constantly reading your emotional state—and that can feel like a deep privacy violation. In a 2023 interview study with 15 US workers, participants described emotion AI as a 'deep privacy violation' over their sensitive emotional information [3]. The persistent nature of a personal AI companion means this isn't a one-time check; it's a continuous surveillance of your inner life, which can erode your sense of safety over time.
The risk isn't just about data leaks—it's about the feeling of being watched and judged. The same study found that workers felt they had to engage in 'emotional labor'—managing their emotions to present a certain face—as a way to preserve privacy over their feelings [3]. In a personal AI companion, this could mean you start performing emotions for the AI, rather than being authentic, which undermines the very purpose of having a companion.
When AI feels too human: threats to your identity and value
A persistent AI companion that mimics human empathy can trigger deep-seated fears about your own uniqueness and worth. In a 2024 study with 410 participants, researchers found that when people faced generative AI's ability to reproduce human cognitive capabilities, they reported significantly higher negative emotions than a control group [1]. These emotions were tied to a perceived threat to 'human identity, uniqueness, and value' [1]. So, the more human-like your AI companion becomes, the more it might make you question what makes you special—a subtle but powerful emotional safety risk.
This threat isn't just a fleeting feeling; it can generalize. The same study showed that negative emotions from one type of AI (generative) extended to a perception of threat from AI technologies as a whole [1]. In a persistent product, this could mean that a frustrating interaction with your AI companion might color your view of all technology, increasing anxiety and distrust in your daily life.
Emotional safety is the foundation—AI can't ignore it
Emotional safety isn't a luxury; it's a prerequisite for well-being, and AI companions must respect it. A 2022 paper in The Lancet Psychiatry argues that emotional and physical safety cannot be separated, and that a lack of emotional safety leads to higher stress and hopelessness [2]. They use examples from psychiatric care, but the principle applies to AI: if an AI companion makes you feel unsafe emotionally, it can't keep you safe in any meaningful way.
The factors that undermine emotional safety are well-known: force majeure, cultural challenges, poor work-rest routines, and communication threats [4]. A persistent AI companion could inadvertently amplify these—for instance, by reminding you of your stress or by failing to understand cultural nuances. The 2021 paper emphasizes that maintaining emotional stability requires personal development and a conscious identity [4]. An AI companion that undermines your sense of self or your coping mechanisms is a direct threat to that stability.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2024, 1 from 2024 or later, 2 in Q1 journals, collectively cited 150 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 74 papers retrieved from a database of over 500 million.
Sources used in this answer
The emotional impact of generative AI: negative emotions and perception of threat
In three studies with 410 participants, exposure to generative AI's human-like abilities increased negative emotions and perceived threats to human identity, uniqueness, and value, generalizing to AI as a whole.
No safety without emotional safety
A 2022 Personal View argues that emotional safety is essential for physical safety, and that controlling behavior without emotional support increases stress and hopelessness, using psychiatric inpatient examples.
Emotion AI at Work: Implications for Workplace Surveillance, Emotional Labor, and Emotional Privacy
An interview study with 15 US workers found that emotion AI in the workplace is seen as a deep privacy violation, enforces emotional labor, and exposes workers to a range of harms, highlighting the need for emotional privacy rights.
EMOTIONAL SAFETY: FACTORS OF RESISTANCE
A 2021 paper identifies factors like force majeure, cultural challenges, and communication threats as impacting emotional safety, and suggests personal development and conscious identity are key to maintaining emotional stability.
Phenomenology-Informed Counseling: Reflections on Emotional Safety and Emotional Initiative in Couple Counseling
A 2023 paper on couple counseling discusses how phenomenological functions like recognition and dialogue foster emotional safety, implying that AI companions would need similar relational stances to be safe.