Large Language Models and Empathy: Systematic Review
Journal of Medical Internet Research · 11.12.2024 · 26 · e52597
DOI: 10.2196/52597 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Journal of Medical Internet Research, online erschienen am 11.12.2024. Verfasst von Vera Sorin und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Mayo Clinic, Rochester, MN, United States. Hauptthemen laut MEDLINE: Empathy. Die Arbeit wurde bisher 260-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Empathy, a fundamental aspect of human interaction, is characterized as the ability to experience another being’s emotions within oneself. In health care, empathy is a fundamental for health care professionals and patients’ interaction. It is a unique quality to humans that large language models (LLMs) are believed to lack.
Objective
We aimed to review the literature on the capacity of LLMs in demonstrating empathy.
Methods
We conducted a literature search on MEDLINE, Google Scholar, PsyArXiv, medRxiv, and arXiv between December 2022 and February 2024. We included English-language full-length publications that evaluated empathy in LLMs’ outputs. We excluded papers evaluating other topics related to emotional intelligence that were not specifically empathy. The included studies’ results, including the LLMs used, performance in empathy tasks, and limitations of the models, along with studies’ metadata were summarized.
Results
A total of 12 studies published in 2023 met the inclusion criteria. ChatGPT-3.5 (OpenAI) was evaluated in all studies, with 6 studies comparing it with other LLMs such GPT-4, LLaMA (Meta), and fine-tuned chatbots. Seven studies focused on empathy within a medical context. The studies reported LLMs to exhibit elements of empathy, including emotions recognition and emotional support in diverse contexts. Evaluation metric included automatic metrics such as Recall-Oriented Understudy for Gisting Evaluation and Bilingual Evaluation Understudy, and human subjective evaluation. Some studies compared performance on empathy with humans, while others compared between different models. In some cases, LLMs were observed to outperform humans in empathy-related tasks. For example, ChatGPT-3.5 was evaluated for its responses to patients’ questions from social media, where ChatGPT’s responses were preferred over those of humans in 78.6% of cases. Other studies used subjective readers’ assigned scores. One study reported a mean empathy score of 1.84-1.9 (scale 0-2) for their fine-tuned LLM, while a different study evaluating ChatGPT-based chatbots reported a mean human rating of 3.43 out of 4 for empathetic responses. Other evaluations were based on the level of the emotional awareness scale, which was reported to be higher for ChatGPT-3.5 than for humans. Another study evaluated ChatGPT and GPT-4 on soft-skills questions in the United States Medical Licensing Examination, where GPT-4 answered 90% of questions correctly. Limitations were noted, including repetitive use of empathic phrases, difficulty following initial instructions, overly lengthy responses, sensitivity to prompts, and overall subjective evaluation metrics influenced by the evaluator’s background.
Conclusions
LLMs exhibit elements of cognitive empathy, recognizing emotions and providing emotionally supportive responses in various contexts. Since social skills are an integral part of intelligence, these advancements bring LLMs closer to human-like interactions and expand their potential use in applications requiring emotional intelligence. However, there remains room for improvement in both the performance of these models and the evaluation strategies used for assessing soft skills.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, Mayo Clinic, Rochester, MN, United States.
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel.
- The Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
- DeepVision Lab, Chaim Sheba Medical Center, Tel Hashomer, Israel.
- Division of Data-Driven and Digital Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
ORCID-Profile: Vera Sorin, Dana Brin, Yiftach Barash, Eli Konen, Alexander Charney, Girish Nadkarni, Eyal Klang
Zitieren
APA
Sorin, V., Brin, D., Barash, Y., Konen, E., Charney, A., Nadkarni, G., & Klang, E. (2024). Large Language Models and Empathy: Systematic Review. Journal of Medical Internet Research, 26, e52597. https://doi.org/10.2196/52597
Vancouver
Sorin V, Brin D, Barash Y, Konen E, Charney A, Nadkarni G, et al. Large Language Models and Empathy: Systematic Review. Journal of Medical Internet Research. 2024;26:e52597. doi:10.2196/52597
BibTeX
@article{Sorin2024,
author = {Sorin, Vera and Brin, Dana and Barash, Yiftach and Konen, Eli and Charney, Alexander and Nadkarni, Girish and Klang, Eyal},
title = {Large Language Models and Empathy: Systematic Review},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e52597},
doi = {10.2196/52597}
}
EUVIMED