Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study
JMIR Formative Research · 2025
DOI: 10.2196/69075 ↗At a glance
Mixed-Methods-Studie in JMIR Formative Research, 2025. Written by Muhammad Hafiz Sulaiman and 2 co-authors. Institution: Department of Mathematical Sciences, Faculty of Science and Technology, National University of Malaysia, Bangi, Malaysia. Major MEDLINE topics: Internet, Hospitals, Private, Patient Satisfaction. Cited 7 times so far (Crossref); 60 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language model (LLM)–assisted content analysis (LACA) is a modification of traditional content analysis, leveraging the LLM to codevelop codebooks and automatically assign thematic codes to a web-based reviews dataset.
Objective
This study aims to develop and validate the use of LACA for analyzing hospital web-based reviews and to identify themes of issues from web-based reviews using this method.
Methods
Web-based reviews for 53 private hospitals in Selangor, Malaysia, were acquired. Fake reviews were filtered out using natural language processing and machine learning algorithms trained on yelp.com validated datasets. GPT-4o mini model application programming interface (API) was then applied to filter out reviews without any quality issues. In total, 200 of the remaining reviews were randomly extracted and fed into the GPT-4o mini model API to produce a codebook validated through parallel human-LLM coding to establish interrater reliability. The codebook was then used to code (label) all reviews in the dataset. The thematic codes were then summarized into themes using factor analysis to increase interpretability.
Results
A total of 14,938 web-based reviews were acquired, of which 1121 (9.3%) were fake, 1279 (12%) contained negative sentiments, and 9635 (88%) did not contain any negative sentiment. GPT-4o mini model subsequently inducted 41 thematic codes together with their definitions. Average human-GPT interrater reliability is perfect (κ=0.81). Factor analysis identified 6 interpretable latent factors: “Service and Communication Effectiveness,” “Clinical Care and Patient Experience,” “Facilities and Amenities Quality,” “Appointment and Patient Flow,” “Financial and Insurance Management,” and “Patient Rights and Accessibility.” The cumulative explained variance for the six factors is 0.74, and Cronbach α is between 0.88 and 0.97 (good and excellent) for all factors except factor 6 (0.61: questionable). The factors identified follow a global pattern of issues identified from the literature.
Conclusions
A data collection and processing pipeline consisting of Python Selenium, the GPT-4o mini model API, and a factor analysis module can support valid and reliable thematic analysis. Despite the potential for collection and information bias in web-based reviews, LACA of web-based reviews is cost-effective, time-efficient, and can be performed in real time, helping hospital managers develop hypotheses for further investigations promptly.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Mathematical Sciences, Faculty of Science and Technology, National University of Malaysia, Bangi, Malaysia.
- Quality Unit, Clinical Management Section, Hospital Sultan Idris Shah, Serdang, Malaysia.
ORCID profiles: Muhammad Hafiz Sulaiman, Nora Muda, Fatimah Abdul Razak
Cite
APA
Sulaiman, M. H., Muda, N., & Abdul Razak, F. (2025). Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study. JMIR Formative Research. https://doi.org/10.2196/69075
Vancouver
Sulaiman MH, Muda N, Abdul Razak F. Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study. JMIR Formative Research. 2025. doi:10.2196/69075
BibTeX
@article{Sulaiman2025,
author = {Sulaiman, Muhammad Hafiz and Muda, Nora and Abdul Razak, Fatimah},
title = {Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/69075}
}
EUVIMED