EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study

Sky Wei Chee Koh, Eunice Rui Ning Wong, John Chong Min Tan, Stephanie C C van der Lubbe, Jun Cong Goh, Ethan Sheng Yong Ching, Ian Wen Yih Chia, Si Hui Low, Ping Young Ang, Queenie Quek, Mehul Motani, Jose M Valderas

Journal of Medical Internet Research · 06.08.2025 · 27 · e74231-e74231

DOI: 10.2196/74231 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 06.08.2025. Verfasst von Sky Wei Chee Koh und 11 weiteren Autor:innen. Beteiligte Einrichtung: Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E …. Hauptthemen laut MEDLINE: Artificial Intelligence, Patient Satisfaction, Language. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Patient complaints provide valuable insights into the performance of health care systems, highlighting potential risks not apparent to staff. Patient complaints can drive systemic changes that enhance patient safety. However, manual categorization and analysis pose a huge logistical challenge, hindering the ability to harness the potential of these data.

Objective

This study aims to evaluate the accuracy of artificial intelligence (AI)–powered categorization of patient complaints in primary care based on the Healthcare Complaint Analysis Tool (HCAT) General Practice (GP) taxonomy and assess the importance of advanced large language models (LLMs) in complaint categorization.

Methods

This cross-sectional study analyzed 1816 anonymous patient complaints from 7 public primary care clinics in Singapore. Complaints were first coded by trained human coders using the HCAT (GP) taxonomy through a rigorous process involving independent assessment and consensus discussions. LLMs (GPT-3.5 turbo, GPT-4o mini, and Claude 3.5 Sonnet) were used to validate manual classification. Claude 3.5 Sonnet was further used to identify complaint themes. LLM classifications were assessed for accuracy and consistency with human coding using accuracy and F 1 -score. Cohen κ and McNemar test evaluated AI-human agreement and compared AI models’ concordance, respectively.

Results

The majority of complaints fell under the HCAT (GP) domain of management (1079/1816, 59.4%), specifically relating to institutional processes (830/1816, 45.7%). Most complaints were of medium severity (994/1816, 54.7%), occurred within the practice (627/1816, 34.5%), and resulted in minimal harm (75.4%). LLMs achieved moderate to good accuracy (58.4%‐95.5%) in HCAT (GP) field classifications, with GPT-4o mini generally outperforming GPT-3.5 turbo, except in severity classification. All 3 LLMs demonstrated moderate concordance rates (average 61.9%‐68.8%) in complaints classification with varying levels of agreement (κ=0.114‐0.623). GPT-4o mini and Claude 3.5 significantly outperformed GPT-3.5 turbo in several fields ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligencePatient SatisfactionLanguageHumansCross-Sectional StudiesPrimary Health CareSingaporeMaleFemaleLarge Language Models

Stichwörter der Autor:innen

artificial intelligencefamily medicinehealth serviceslarge language modelspatient complaintsprimary care

Autor:innen und Institutionen

  • Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E Kent Ridge Road, Singapore, 119228, Singapore, 65 67163185.
  • National University Polyclinics, National University Health System, Singapore, Singapore.
  • Department of Family Medicine, National University Health System, Singapore, Singapore.
  • Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore, Singapore, Singapore.
  • Institute of Data Science, N.1 Institute for Health, Institute for Digital Medicine, National University of Singapore, Singapore, Singapore.
  • Centre for Research in Health Systems Performance and Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.

Zitieren

APA

Koh, S. W. C., Wong, E. R. N., Tan, J. C. M., van der Lubbe, S. C. C., Goh, J. C., Ching, E. S. Y., Chia, I. W. Y., Low, S. H., Ang, P. Y., Quek, Q., Motani, M., & Valderas, J. M. (2025). Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research, 27, e74231-e74231. https://doi.org/10.2196/74231

Vancouver

Koh SWC, Wong ERN, Tan JCM, van der Lubbe SCC, Goh JC, Ching ESY, et al. Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research. 2025;27:e74231-e74231. doi:10.2196/74231

BibTeX
@article{Koh2025,
  author = {Koh, Sky Wei Chee and Wong, Eunice Rui Ning and Tan, John Chong Min and van der Lubbe, Stephanie C C and Goh, Jun Cong and Ching, Ethan Sheng Yong and Chia, Ian Wen Yih and Low, Si Hui and Ang, Ping Young and Quek, Queenie and Motani, Mehul and Valderas, Jose M},
  title = {Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e74231--e74231},
  doi = {10.2196/74231}
}
Kostenlos starten →