Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey
Journal of Medical Internet Research · 09.08.2024 · 26 · e56413
DOI: 10.2196/56413 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 09.08.2024. Verfasst von Lorraine Pei Xian Yong und 4 weiteren Autor:innen. Beteiligte Einrichtung: Emergency Medicine Department, National University Hospital, National University Health System, Singapore, Singapore. Hauptthemen laut MEDLINE: Patient Satisfaction. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Patient complaints are a perennial challenge faced by health care institutions globally, requiring extensive time and effort from health care workers. Despite these efforts, patient dissatisfaction remains high. Recent studies on the use of large language models (LLMs) such as the GPT models developed by OpenAI in the health care sector have shown great promise, with the ability to provide more detailed and empathetic responses as compared to physicians. LLMs could potentially be used in responding to patient complaints to improve patient satisfaction and complaint response time.
Objective
This study aims to evaluate the performance of LLMs in addressing patient complaints received by a tertiary health care institution, with the goal of enhancing patient satisfaction.
Methods
Anonymized patient complaint emails and associated responses from the patient relations department were obtained. ChatGPT-4.0 (OpenAI, Inc) was provided with the same complaint email and tasked to generate a response. The complaints and the respective responses were uploaded onto a web-based questionnaire. Respondents were asked to rate both responses on a 10-point Likert scale for 4 items: appropriateness, completeness, empathy, and satisfaction. Participants were also asked to choose a preferred response at the end of each scenario.
Results
There was a total of 188 respondents, of which 115 (61.2%) were health care workers. A majority of the respondents, including both health care and non–health care workers, preferred replies from ChatGPT (n=164, 87.2% to n=183, 97.3%). GPT-4.0 responses were rated higher in all 4 assessed items with all median scores of 8 (IQR 7-9) compared to human responses (appropriateness 5, IQR 3-7; empathy 4, IQR 3-6; quality 5, IQR 3-6; satisfaction 5, IQR 3-6; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Emergency Medicine Department, National University Hospital, National University Health System, Singapore, Singapore.
- Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
- Urgent Care Centre, Alexandra Hospital, National University Health System, Singapore, Singapore.
- Department of Urology, Singapore General Hospital, Singapore, Singapore.
ORCID-Profile: Lorraine Pei Xian Yong, Joshua Yi Min Tung, Zi Yao Lee, Win Sen Kuan, Mui Teng Chua
Zitieren
APA
Yong, L. P. X., Tung, J. Y. M., Lee, Z. Y., Kuan, W. S., & Chua, M. T. (2024). Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey. Journal of Medical Internet Research, 26, e56413. https://doi.org/10.2196/56413
Vancouver
Yong LPX, Tung JYM, Lee ZY, Kuan WS, Chua MT. Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey. Journal of Medical Internet Research. 2024;26:e56413. doi:10.2196/56413
BibTeX
@article{Yong2024,
author = {Yong, Lorraine Pei Xian and Tung, Joshua Yi Min and Lee, Zi Yao and Kuan, Win Sen and Chua, Mui Teng},
title = {Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e56413},
doi = {10.2196/56413}
}
EUVIMED