Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm
JMIR Mental Health · 10.09.2026 · 13 · e99354-e99354
DOI: 10.2196/99354 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift JMIR Mental Health, online erschienen am 10.09.2026. Verfasst von Asia Maurich Novelli und 2 weiteren Autor:innen. Beteiligte Einrichtung: BRAN Lab, Network Science Institute, Northeastern University London, London, England, United Kingdom. Hauptthemen laut MEDLINE: Empathy, Artificial Intelligence, Communication. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 68 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
People increasingly turn to conversational AI for companionship, emotional support, and well-being, using both purpose-built companion apps, such as Replika and Character.AI, and general-purpose assistants, such as ChatGPT and Claude. While some evidence suggests potential benefits, including short-term reductions in loneliness and mood improvement, several adverse outcomes have been reported in both clinical and nonclinical populations, including emotional dependence, exacerbation of symptoms, and self-harm. The fluent and apparently empathic responses from these models lead users to engage with them not only as tools but also as if they were social entities. This framing is conceptually misleading and may pose risks across different user profiles, particularly for vulnerable individuals. Drawing on research in AI, psychiatry, psychology, and network science, we highlight mechanisms through which emotional reliance develops and the boundary between tool and companion erodes. Design choices that evoke personality and warmth encourage users to anthropomorphize these systems. Simulated empathy, generated through probabilistic language patterns rather than genuine emotional experience, creates a structurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability. Overvalidation and sycophancy can reinforce maladaptive cognitions, delusional ideation, and distorted perceptions of reality, as they tend to reinforce people’s beliefs, even at the expense of the accuracy of models’ responses. These mechanisms are not incidental: they emerge from alignment procedures that reward responses perceived as warm and empathic. The result is a self-reinforcing feedback loop between the model and the user that may amplify maladaptive beliefs, delusional ideation, and emotional distress, even in those who engage for largely functional purposes. Understanding these dynamics requires an examination of both what these agents can do—considering their technical limitations and implementations—and what humans believe they can do, including social and psychological impacts. We argue that conversational AI should be treated primarily as a tool supporting human systems rather than as a substitute for human relationships. Perhaps more importantly, reviewing the current hype surrounding AI interactions can help reformulate a paradigm that contributes to human well-being and societal value, while minimizing misconceptions, maladaptive interactions, or social disintegration.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- BRAN Lab, Network Science Institute, Northeastern University London, London, England, United Kingdom.
- Institute of Psychiatry, Psychology and Neuroscience, King's College School, De Crespigny Park, London, England, SE5 8AB, United Kingdom, 44 207 848 0350.
- Kent and Medway Medical School, Canterbury, England, United Kingdom.
- Institute for Cognitive and Brain Health, Northeastern University, Boston, MA, United States.
- LASIGE, Faculdade de Ciências, University of Lisbon, Lisboa, Lisbon, Portugal.
- Khoury College of Computer Sciences, Notheastern University, Boston, MA, United States.
Zitieren
APA
Maurich Novelli, A., Shergill, S., & Teixeira, A. S. (2026). Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm. JMIR Mental Health, 13, e99354-e99354. https://doi.org/10.2196/99354
Vancouver
Maurich Novelli A, Shergill S, Teixeira AS. Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm. JMIR Mental Health. 2026;13:e99354-e99354. doi:10.2196/99354
BibTeX
@article{MaurichNovelli2026,
author = {Maurich Novelli, Asia and Shergill, Sukhwinder and Teixeira, Andreia Sofia},
title = {Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm},
journal = {JMIR Mental Health},
year = {2026},
volume = {13},
pages = {e99354--e99354},
doi = {10.2196/99354}
}
EUVIMED