Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study
Journal of Medical Internet Research · 2025-08-06 · 27 · e74231-e74231
DOI: 10.2196/74231 ↗At a glance
Querschnittstudie in Journal of Medical Internet Research, published online on 2025-08-06. Written by Sky Wei Chee Koh and 11 co-authors. Institution: Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E …. Major MEDLINE topics: Artificial Intelligence, Patient Satisfaction, Language. Cited 12 times so far (Crossref); 35 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Patient complaints provide valuable insights into the performance of health care systems, highlighting potential risks not apparent to staff. Patient complaints can drive systemic changes that enhance patient safety. However, manual categorization and analysis pose a huge logistical challenge, hindering the ability to harness the potential of these data.
Objective
This study aims to evaluate the accuracy of artificial intelligence (AI)–powered categorization of patient complaints in primary care based on the Healthcare Complaint Analysis Tool (HCAT) General Practice (GP) taxonomy and assess the importance of advanced large language models (LLMs) in complaint categorization.
Methods
This cross-sectional study analyzed 1816 anonymous patient complaints from 7 public primary care clinics in Singapore. Complaints were first coded by trained human coders using the HCAT (GP) taxonomy through a rigorous process involving independent assessment and consensus discussions. LLMs (GPT-3.5 turbo, GPT-4o mini, and Claude 3.5 Sonnet) were used to validate manual classification. Claude 3.5 Sonnet was further used to identify complaint themes. LLM classifications were assessed for accuracy and consistency with human coding using accuracy and F 1 -score. Cohen κ and McNemar test evaluated AI-human agreement and compared AI models’ concordance, respectively.
Results
The majority of complaints fell under the HCAT (GP) domain of management (1079/1816, 59.4%), specifically relating to institutional processes (830/1816, 45.7%). Most complaints were of medium severity (994/1816, 54.7%), occurred within the practice (627/1816, 34.5%), and resulted in minimal harm (75.4%). LLMs achieved moderate to good accuracy (58.4%‐95.5%) in HCAT (GP) field classifications, with GPT-4o mini generally outperforming GPT-3.5 turbo, except in severity classification. All 3 LLMs demonstrated moderate concordance rates (average 61.9%‐68.8%) in complaints classification with varying levels of agreement (κ=0.114‐0.623). GPT-4o mini and Claude 3.5 significantly outperformed GPT-3.5 turbo in several fields ( P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E Kent Ridge Road, Singapore, 119228, Singapore, 65 67163185.
- National University Polyclinics, National University Health System, Singapore, Singapore.
- Department of Family Medicine, National University Health System, Singapore, Singapore.
- Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore, Singapore, Singapore.
- Institute of Data Science, N.1 Institute for Health, Institute for Digital Medicine, National University of Singapore, Singapore, Singapore.
- Centre for Research in Health Systems Performance and Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
Cite
APA
Koh, S. W. C., Wong, E. R. N., Tan, J. C. M., van der Lubbe, S. C. C., Goh, J. C., Ching, E. S. Y., Chia, I. W. Y., Low, S. H., Ang, P. Y., Quek, Q., Motani, M., & Valderas, J. M. (2025). Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research, 27, e74231-e74231. https://doi.org/10.2196/74231
Vancouver
Koh SWC, Wong ERN, Tan JCM, van der Lubbe SCC, Goh JC, Ching ESY, et al. Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research. 2025;27:e74231-e74231. doi:10.2196/74231
BibTeX
@article{Koh2025,
author = {Koh, Sky Wei Chee and Wong, Eunice Rui Ning and Tan, John Chong Min and van der Lubbe, Stephanie C C and Goh, Jun Cong and Ching, Ethan Sheng Yong and Chia, Ian Wen Yih and Low, Si Hui and Ang, Ping Young and Quek, Queenie and Motani, Mehul and Valderas, Jose M},
title = {Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e74231--e74231},
doi = {10.2196/74231}
}
EUVIMED