EUVIMEDObjevte EUVIMED
QuerschnittstudieOpen access

Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey

Lorraine Pei Xian Yong iD, Joshua Yi Min Tung iD, Zi Yao Lee iD, Win Sen Kuan iD, Mui Teng Chua iD

Journal of Medical Internet Research · 2024-08-09 · 26 · e56413

DOI: 10.2196/56413 ↗

At a glance

Querschnittstudie in Journal of Medical Internet Research, published online on 2024-08-09. Written by Lorraine Pei Xian Yong and 4 co-authors. Institution: Emergency Medicine Department, National University Hospital, National University Health System, Singapore, Singapore. Major MEDLINE topics: Patient Satisfaction. Cited 16 times so far (Crossref); 33 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Patient complaints are a perennial challenge faced by health care institutions globally, requiring extensive time and effort from health care workers. Despite these efforts, patient dissatisfaction remains high. Recent studies on the use of large language models (LLMs) such as the GPT models developed by OpenAI in the health care sector have shown great promise, with the ability to provide more detailed and empathetic responses as compared to physicians. LLMs could potentially be used in responding to patient complaints to improve patient satisfaction and complaint response time.

Objective

This study aims to evaluate the performance of LLMs in addressing patient complaints received by a tertiary health care institution, with the goal of enhancing patient satisfaction.

Methods

Anonymized patient complaint emails and associated responses from the patient relations department were obtained. ChatGPT-4.0 (OpenAI, Inc) was provided with the same complaint email and tasked to generate a response. The complaints and the respective responses were uploaded onto a web-based questionnaire. Respondents were asked to rate both responses on a 10-point Likert scale for 4 items: appropriateness, completeness, empathy, and satisfaction. Participants were also asked to choose a preferred response at the end of each scenario.

Results

There was a total of 188 respondents, of which 115 (61.2%) were health care workers. A majority of the respondents, including both health care and non–health care workers, preferred replies from ChatGPT (n=164, 87.2% to n=183, 97.3%). GPT-4.0 responses were rated higher in all 4 assessed items with all median scores of 8 (IQR 7-9) compared to human responses (appropriateness 5, IQR 3-7; empathy 4, IQR 3-6; quality 5, IQR 3-6; satisfaction 5, IQR 3-6; P

Subjects

MeSH terms (MEDLINE)

Patient SatisfactionHumansCross-Sectional StudiesFemaleSurveys and QuestionnairesMaleAdultInternetLanguageMiddle AgedElectronic Mail

Author keywords

ChatGPTartificial intelligenceefficiencyempathyhealth care complaintlarge language modelspatient complaintpatient satisfactionresource allocation

Authors and institutions

  • Emergency Medicine Department, National University Hospital, National University Health System, Singapore, Singapore.
  • Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
  • Urgent Care Centre, Alexandra Hospital, National University Health System, Singapore, Singapore.
  • Department of Urology, Singapore General Hospital, Singapore, Singapore.

ORCID profiles: Lorraine Pei Xian Yong, Joshua Yi Min Tung, Zi Yao Lee, Win Sen Kuan, Mui Teng Chua

Cite

APA

Yong, L. P. X., Tung, J. Y. M., Lee, Z. Y., Kuan, W. S., & Chua, M. T. (2024). Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey. Journal of Medical Internet Research, 26, e56413. https://doi.org/10.2196/56413

Vancouver

Yong LPX, Tung JYM, Lee ZY, Kuan WS, Chua MT. Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey. Journal of Medical Internet Research. 2024;26:e56413. doi:10.2196/56413

BibTeX
@article{Yong2024,
  author = {Yong, Lorraine Pei Xian and Tung, Joshua Yi Min and Lee, Zi Yao and Kuan, Win Sen and Chua, Mui Teng},
  title = {Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e56413},
  doi = {10.2196/56413}
}
Start for free →