EUVIMEDScopri EUVIMED
QuerschnittstudieOpen access

Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study

Sky Wei Chee Koh, Eunice Rui Ning Wong, John Chong Min Tan, Stephanie C C van der Lubbe, Jun Cong Goh, Ethan Sheng Yong Ching, Ian Wen Yih Chia, Si Hui Low, Ping Young Ang, Queenie Quek, Mehul Motani, Jose M Valderas

Journal of Medical Internet Research · 2025-08-06 · 27 · e74231-e74231

DOI: 10.2196/74231 ↗

At a glance

Querschnittstudie in Journal of Medical Internet Research, published online on 2025-08-06. Written by Sky Wei Chee Koh and 11 co-authors. Institution: Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E …. Major MEDLINE topics: Artificial Intelligence, Patient Satisfaction, Language. Cited 12 times so far (Crossref); 35 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Patient complaints provide valuable insights into the performance of health care systems, highlighting potential risks not apparent to staff. Patient complaints can drive systemic changes that enhance patient safety. However, manual categorization and analysis pose a huge logistical challenge, hindering the ability to harness the potential of these data.

Objective

This study aims to evaluate the accuracy of artificial intelligence (AI)–powered categorization of patient complaints in primary care based on the Healthcare Complaint Analysis Tool (HCAT) General Practice (GP) taxonomy and assess the importance of advanced large language models (LLMs) in complaint categorization.

Methods

This cross-sectional study analyzed 1816 anonymous patient complaints from 7 public primary care clinics in Singapore. Complaints were first coded by trained human coders using the HCAT (GP) taxonomy through a rigorous process involving independent assessment and consensus discussions. LLMs (GPT-3.5 turbo, GPT-4o mini, and Claude 3.5 Sonnet) were used to validate manual classification. Claude 3.5 Sonnet was further used to identify complaint themes. LLM classifications were assessed for accuracy and consistency with human coding using accuracy and F 1 -score. Cohen κ and McNemar test evaluated AI-human agreement and compared AI models’ concordance, respectively.

Results

The majority of complaints fell under the HCAT (GP) domain of management (1079/1816, 59.4%), specifically relating to institutional processes (830/1816, 45.7%). Most complaints were of medium severity (994/1816, 54.7%), occurred within the practice (627/1816, 34.5%), and resulted in minimal harm (75.4%). LLMs achieved moderate to good accuracy (58.4%‐95.5%) in HCAT (GP) field classifications, with GPT-4o mini generally outperforming GPT-3.5 turbo, except in severity classification. All 3 LLMs demonstrated moderate concordance rates (average 61.9%‐68.8%) in complaints classification with varying levels of agreement (κ=0.114‐0.623). GPT-4o mini and Claude 3.5 significantly outperformed GPT-3.5 turbo in several fields ( P

Subjects

MeSH terms (MEDLINE)

Artificial IntelligencePatient SatisfactionLanguageHumansCross-Sectional StudiesPrimary Health CareSingaporeMaleFemaleLarge Language Models

Author keywords

artificial intelligencefamily medicinehealth serviceslarge language modelspatient complaintsprimary care

Authors and institutions

  • Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E Kent Ridge Road, Singapore, 119228, Singapore, 65 67163185.
  • National University Polyclinics, National University Health System, Singapore, Singapore.
  • Department of Family Medicine, National University Health System, Singapore, Singapore.
  • Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore, Singapore, Singapore.
  • Institute of Data Science, N.1 Institute for Health, Institute for Digital Medicine, National University of Singapore, Singapore, Singapore.
  • Centre for Research in Health Systems Performance and Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.

Cite

APA

Koh, S. W. C., Wong, E. R. N., Tan, J. C. M., van der Lubbe, S. C. C., Goh, J. C., Ching, E. S. Y., Chia, I. W. Y., Low, S. H., Ang, P. Y., Quek, Q., Motani, M., & Valderas, J. M. (2025). Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research, 27, e74231-e74231. https://doi.org/10.2196/74231

Vancouver

Koh SWC, Wong ERN, Tan JCM, van der Lubbe SCC, Goh JC, Ching ESY, et al. Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study. Journal of Medical Internet Research. 2025;27:e74231-e74231. doi:10.2196/74231

BibTeX
@article{Koh2025,
  author = {Koh, Sky Wei Chee and Wong, Eunice Rui Ning and Tan, John Chong Min and van der Lubbe, Stephanie C C and Goh, Jun Cong and Ching, Ethan Sheng Yong and Chia, Ian Wen Yih and Low, Si Hui and Ang, Ping Young and Quek, Queenie and Motani, Mehul and Valderas, Jose M},
  title = {Classifying Patient Complaints Using Artificial Intelligence–Powered Large Language Models: Cross-Sectional Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e74231--e74231},
  doi = {10.2196/74231}
}
Start for free →