EUVIMEDSkopri EUVIMED
VergleichsstudieMixed-Methods-StudieOpen access

A Comparison of Responses from Human Therapists and Large Language Model–Based Chatbots to Assess Therapeutic Communication: Mixed Methods Study

Till Scholich iD, Maya Barr iD, Shannon Wiltsey Stirman iD, Shriti Raj iD

JMIR Mental Health · 2025-05-21 · 12 · e69709

DOI: 10.2196/69709 ↗

At a glance

Vergleichsstudie in JMIR Mental Health, published online on 2025-05-21. Written by Till Scholich and 3 co-authors. Institution: Institute for Human-Centered AI, Stanford University, Stanford, CA, United States. Major MEDLINE topics: Communication, Professional-Patient Relations, Psychotherapy, Attitude of Health Personnel. Cited 85 times so far (Crossref); 42 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Consumers are increasingly using large language model–based chatbots to seek mental health advice or intervention due to ease of access and limited availability of mental health professionals. However, their suitability and safety for mental health applications remain underexplored, particularly in comparison to professional therapeutic practices.

Objective

This study aimed to evaluate how general-purpose chatbots respond to mental health scenarios and compare their responses to those provided by licensed therapists. Specifically, we sought to identify chatbots’ strengths and limitations, as well as the ethical and practical considerations necessary for their use in mental health care.

Methods

We conducted a mixed methods study to compare responses from chatbots and licensed therapists to scripted mental health scenarios. We created 2 fictional scenarios and prompted 3 chatbots to create 6 interaction logs. We recruited 17 therapists and conducted study sessions that consisted of 3 activities. First, therapists responded to the 2 scenarios using a Qualtrics form. Second, therapists went through the 6 interaction logs using a think-aloud procedure to highlight their thoughts about the chatbots’ responses. Finally, we conducted a semistructured interview to explore subjective opinions on the use of chatbots for supporting mental health. The study sessions were analyzed using thematic analysis. The interaction logs from chatbot and therapist responses were coded using the Multitheoretical List of Therapeutic Interventions codes and then compared to each other.

Results

We identified 7 themes describing the strengths and limitations of the chatbots as compared to therapists. These include elements of good therapy in chatbot responses, conversational style of chatbots, insufficient inquiry and feedback seeking by chatbots, chatbot interventions, client engagement, chatbots’ responses to crisis situations, and considerations for chatbot-based therapy. In the use of Multitheoretical List of Therapeutic Interventions codes, we found that therapists evoked more elaboration (Mann-Whitney U=9; P=.001) and used more self-disclosure (U=45.5; P=.37) as compared to the chatbots. The chatbots used affirming (U=28; P=.045) and reassuring (U=23; P=.02) language more often than the therapists. The chatbots also used psychoeducation (U=22.5; P=.02) and suggestions (U=12.5; P=.003) more often than the therapists.

Conclusions

Our study demonstrates the unsuitability of general-purpose chatbots to safely engage in mental health conversations, particularly in crisis situations. While chatbots display elements of good therapy, such as validation and reassurance, overuse of directive advice without sufficient inquiry and use of generic interventions make them unsuitable as therapeutic agents. Careful research and evaluation will be necessary to determine the impact of chatbot interactions and to identify the most appropriate use cases related to mental health.

Subjects

MeSH terms (MEDLINE)

CommunicationProfessional-Patient RelationsPsychotherapyAttitude of Health PersonnelMental DisordersHealth PersonnelHumansFemaleMaleAdultMiddle AgedLarge Language ModelsGenerative Artificial Intelligence

Author keywords

AIAI therapyLLMartificial intelligenceartificial intelligence therapylarge language modellarge language modelsmental healththerapists

Authors and institutions

  • Institute for Human-Centered AI, Stanford University, Stanford, CA, United States.
  • PGSP-Stanford PsyD Consortium, Palo Alto University, Palo Alto, CA, United States.
  • Dissemination and Training Division, National Center for PTSD, Menlo Park, CA, United States.
  • Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, United States.
  • Department of Medicine Center for Biomedical Informatics Research, Stanford University, Stanford, CA, United States.

ORCID profiles: Till Scholich, Maya Barr, Shannon Wiltsey Stirman, Shriti Raj

Cite

APA

Scholich, T., Barr, M., Wiltsey Stirman, S., & Raj, S. (2025). A Comparison of Responses from Human Therapists and Large Language Model–Based Chatbots to Assess Therapeutic Communication: Mixed Methods Study. JMIR Mental Health, 12, e69709. https://doi.org/10.2196/69709

Vancouver

Scholich T, Barr M, Wiltsey Stirman S, Raj S. A Comparison of Responses from Human Therapists and Large Language Model–Based Chatbots to Assess Therapeutic Communication: Mixed Methods Study. JMIR Mental Health. 2025;12:e69709. doi:10.2196/69709

BibTeX
@article{Scholich2025,
  author = {Scholich, Till and Barr, Maya and Wiltsey Stirman, Shannon and Raj, Shriti},
  title = {A Comparison of Responses from Human Therapists and Large Language Model–Based Chatbots to Assess Therapeutic Communication: Mixed Methods Study},
  journal = {JMIR Mental Health},
  year = {2025},
  volume = {12},
  pages = {e69709},
  doi = {10.2196/69709}
}
Start for free →