EUVIMEDScopri EUVIMED
Mixed-Methods-StudieOpen access

Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study

Muhammad Hafiz Sulaiman iD, Nora Muda iD, Fatimah Abdul Razak iD

JMIR Formative Research · 2025

DOI: 10.2196/69075 ↗

At a glance

Mixed-Methods-Studie in JMIR Formative Research, 2025. Written by Muhammad Hafiz Sulaiman and 2 co-authors. Institution: Department of Mathematical Sciences, Faculty of Science and Technology, National University of Malaysia, Bangi, Malaysia. Major MEDLINE topics: Internet, Hospitals, Private, Patient Satisfaction. Cited 7 times so far (Crossref); 60 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Large language model (LLM)–assisted content analysis (LACA) is a modification of traditional content analysis, leveraging the LLM to codevelop codebooks and automatically assign thematic codes to a web-based reviews dataset.

Objective

This study aims to develop and validate the use of LACA for analyzing hospital web-based reviews and to identify themes of issues from web-based reviews using this method.

Methods

Web-based reviews for 53 private hospitals in Selangor, Malaysia, were acquired. Fake reviews were filtered out using natural language processing and machine learning algorithms trained on yelp.com validated datasets. GPT-4o mini model application programming interface (API) was then applied to filter out reviews without any quality issues. In total, 200 of the remaining reviews were randomly extracted and fed into the GPT-4o mini model API to produce a codebook validated through parallel human-LLM coding to establish interrater reliability. The codebook was then used to code (label) all reviews in the dataset. The thematic codes were then summarized into themes using factor analysis to increase interpretability.

Results

A total of 14,938 web-based reviews were acquired, of which 1121 (9.3%) were fake, 1279 (12%) contained negative sentiments, and 9635 (88%) did not contain any negative sentiment. GPT-4o mini model subsequently inducted 41 thematic codes together with their definitions. Average human-GPT interrater reliability is perfect (κ=0.81). Factor analysis identified 6 interpretable latent factors: “Service and Communication Effectiveness,” “Clinical Care and Patient Experience,” “Facilities and Amenities Quality,” “Appointment and Patient Flow,” “Financial and Insurance Management,” and “Patient Rights and Accessibility.” The cumulative explained variance for the six factors is 0.74, and Cronbach α is between 0.88 and 0.97 (good and excellent) for all factors except factor 6 (0.61: questionable). The factors identified follow a global pattern of issues identified from the literature.

Conclusions

A data collection and processing pipeline consisting of Python Selenium, the GPT-4o mini model API, and a factor analysis module can support valid and reliable thematic analysis. Despite the potential for collection and information bias in web-based reviews, LACA of web-based reviews is cost-effective, time-efficient, and can be performed in real time, helping hospital managers develop hypotheses for further investigations promptly.

Subjects

MeSH terms (MEDLINE)

InternetHospitals, PrivatePatient SatisfactionMalaysiaHumansReproducibility of ResultsNatural Language ProcessingLarge Language Models

Author keywords

big datahospital qualitylarge language modelpatient satisfactionweb-based review

Authors and institutions

  • Department of Mathematical Sciences, Faculty of Science and Technology, National University of Malaysia, Bangi, Malaysia.
  • Quality Unit, Clinical Management Section, Hospital Sultan Idris Shah, Serdang, Malaysia.

ORCID profiles: Muhammad Hafiz Sulaiman, Nora Muda, Fatimah Abdul Razak

Cite

APA

Sulaiman, M. H., Muda, N., & Abdul Razak, F. (2025). Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study. JMIR Formative Research. https://doi.org/10.2196/69075

Vancouver

Sulaiman MH, Muda N, Abdul Razak F. Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study. JMIR Formative Research. 2025. doi:10.2196/69075

BibTeX
@article{Sulaiman2025,
  author = {Sulaiman, Muhammad Hafiz and Muda, Nora and Abdul Razak, Fatimah},
  title = {Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language Model–Assisted Content Analysis: Mixed Methods Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/69075}
}
Start for free →