EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Classification of Patients’ Judgments of Their Physicians in Web-Based Written Reviews Using Natural Language Processing: Algorithm Development and Validation

Farrah Madanay iD, Karissa Tu iD, Ada Campagna iD, J Kelly Davis iD, Steven S Doerstling iD, Felicia Chen iD, Peter A Ubel iD

Journal of Medical Internet Research · 01.08.2024 · 26 · e50236

DOI: 10.2196/50236 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 01.08.2024. Verfasst von Farrah Madanay und 6 weiteren Autor:innen. Beteiligte Einrichtung: Sanford School of Public Policy, Duke University, Durham, NC, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Algorithms, Internet. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Patients increasingly rely on web-based physician reviews to choose a physician and share their experiences. However, the unstructured text of these written reviews presents a challenge for researchers seeking to make inferences about patients’ judgments. Methods previously used to identify patient judgments within reviews, such as hand-coding and dictionary-based approaches, have posed limitations to sample size and classification accuracy. Advanced natural language processing methods can help overcome these limitations and promote further analysis of physician reviews on these popular platforms.

Objective

This study aims to train, test, and validate an advanced natural language processing algorithm for classifying the presence and valence of 2 dimensions of patient judgments in web-based physician reviews: interpersonal manner and technical competence.

Methods

We sampled 345,053 reviews for 167,150 physicians across the United States from Healthgrades.com, a commercial web-based physician rating and review website. We hand-coded 2000 written reviews and used those reviews to train and test a transformer classification algorithm called the Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach (RoBERTa). The 2 fine-tuned models coded the reviews for the presence and positive or negative valence of patients’ interpersonal manner or technical competence judgments of their physicians. We evaluated the performance of the 2 models against 200 hand-coded reviews and validated the models using the full sample of 345,053 RoBERTa-coded reviews.

Results

The interpersonal manner model was 90% accurate with precision of 0.89, recall of 0.90, and weighted F1-score of 0.89. The technical competence model was 90% accurate with precision of 0.91, recall of 0.90, and weighted F1-score of 0.90. Positive-valence judgments were associated with higher review star ratings whereas negative-valence judgments were associated with lower star ratings. Analysis of the data by review rating and physician gender corresponded with findings in prior literature.

Conclusions

Our 2 classification models coded interpersonal manner and technical competence judgments with high precision, recall, and accuracy. These models were validated using review star ratings and results from previous research. RoBERTa can accurately classify unstructured, web-based review text at scale. Future work could explore the use of this algorithm with other textual data, such as social media posts and electronic health records.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingAlgorithmsInternetHumansFemaleMalePhysiciansPhysician-Patient RelationsJudgmentAdultMiddle Aged

Stichwörter der Autor:innen

RoBERTahealthcare qualitymachine learningnatural language processingpatient carepatient experiencepatient judgmentspatient-authored reviewspsychologytext classificationweb-based physician reviews

Autor:innen und Institutionen

  • Sanford School of Public Policy, Duke University, Durham, NC, United States.
  • Center for Bioethics and Social Sciences in Medicine, University of Michigan Medical School, Ann Arbor, MI, United States.
  • Fuqua School of Business, Duke University, Durham, NC, United States.
  • School of Medicine, University of Washington, Seattle, WA, United States.
  • Center for Advanced Hindsight, Duke University, Durham, NC, United States.
  • Department of Sociology, University of California, Los Angeles, Los Angeles, CA, United States.

ORCID-Profile: Farrah Madanay, Karissa Tu, Ada Campagna, J Kelly Davis, Steven S Doerstling, Felicia Chen, Peter A Ubel

Zitieren

APA

Madanay, F., Tu, K., Campagna, A., Davis, J. K., Doerstling, S. S., Chen, F., & Ubel, P. A. (2024). Classification of Patients’ Judgments of Their Physicians in Web-Based Written Reviews Using Natural Language Processing: Algorithm Development and Validation. Journal of Medical Internet Research, 26, e50236. https://doi.org/10.2196/50236

Vancouver

Madanay F, Tu K, Campagna A, Davis JK, Doerstling SS, Chen F, et al. Classification of Patients’ Judgments of Their Physicians in Web-Based Written Reviews Using Natural Language Processing: Algorithm Development and Validation. Journal of Medical Internet Research. 2024;26:e50236. doi:10.2196/50236

BibTeX
@article{Madanay2024,
  author = {Madanay, Farrah and Tu, Karissa and Campagna, Ada and Davis, J Kelly and Doerstling, Steven S and Chen, Felicia and Ubel, Peter A},
  title = {Classification of Patients’ Judgments of Their Physicians in Web-Based Written Reviews Using Natural Language Processing: Algorithm Development and Validation},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e50236},
  doi = {10.2196/50236}
}
Kostenlos starten →