EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access · CC BY 4.0

Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis

Rebecka Maria Norman iD, Lilja Charlotte Storset, Petter Mæhlum, Hilde Hestad Iversen iD, Elma Jelin, Erik Velldal, Lilja Øvrelid, Øyvind Bjertnæs iD

Family Practice · 31.07.2026 · 43 (5)

DOI: 10.1093/fampra/cmag071 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Family Practice, online erschienen am 09.09.2026. Verfasst von Rebecka Maria Norman und 7 weiteren Autor:innen. Beteiligte Einrichtung: Division for Health Services, Norwegian Institute of Public Health, N-0213 Oslo, Norway. Hauptthemen laut MEDLINE: General Practice, Patient Satisfaction. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Norwegian Research Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Automated sentiment analysis can be used to analyse free-text patient experience comments, but in health services research, it requires technical performance and methodological validity. Traditional models lack contextual understanding, whereas masked language models (MLMs) enable nuanced sentiment classification. This study is the first to fine-tune an MLM (NorBERT3large) on human-annotated data and evaluate the validity of four-category sentiment classifications across survey and online feedback in general practice. Objective(s) To evaluate a fine-tuned MLM against human-annotated sentiment labels, compare its performance with traditional methods and a large language model, and assess convergent and known-groups validity using survey and online comments.

Methods

NorBERT3large was fine-tuned on human-annotated free-text survey comments. Performance was evaluated using F1-scores. The model classified unseen online data. Construct validity was assessed per COSMIN guidelines through correlations with patient experience scores and subgroup analyses.

Results

The fine-tuned NorBERT3large achieved high F1-scores on held-out survey test data and outperformed traditional models used in prior patient experience research, as well as a zero-shot large language model reference. On unseen online reviews, performance was high in the random sample and strongest for mixed and positive sentiment in the balanced evaluation set, while the rare neutral comments were difficult to classify reliably. Construct validity was supported by expected correlations with patient experience scores and known subgroup differences.

Conclusion

Beyond good classification performance, the findings support automated sentiment analysis as a valid patient experience measure in general practice. The sentiment classifications may complement traditional patient experience measures and help identify areas for quality improvement.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

General PracticePatient SatisfactionHumansLarge Language ModelsSurveys and QuestionnairesReproducibility of Results

Stichwörter der Autor:innen

general practicehealth care surveysmachine learningpatient-centred careprimary health carevalidation study

Autor:innen und Institutionen

  • Division for Health Services, Norwegian Institute of Public Health, N-0213 Oslo, Norway.
  • Department of Informatics, Section for Machine Learning, Language Technology Group, University of Oslo, N-0316 Oslo, Norway.

ORCID-Profile: Rebecka Maria Norman, Hilde Hestad Iversen, Øyvind Bjertnæs

Förderung

  • Norwegian Research Council

Zitieren

APA

Norman, R. M., Storset, L. C., Mæhlum, P., Iversen, H. H., Jelin, E., Velldal, E., Øvrelid, L., & Bjertnæs, Ø. (2026). Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis. Family Practice, 43(5). https://doi.org/10.1093/fampra/cmag071

Vancouver

Norman RM, Storset LC, Mæhlum P, Iversen HH, Jelin E, Velldal E, et al. Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis. Family Practice. 2026;43(5). doi:10.1093/fampra/cmag071

BibTeX
@article{Norman2026,
  author = {Norman, Rebecka Maria and Storset, Lilja Charlotte and Mæhlum, Petter and Iversen, Hilde Hestad and Jelin, Elma and Velldal, Erik and Øvrelid, Lilja and Bjertnæs, Øyvind},
  title = {Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis},
  journal = {Family Practice},
  year = {2026},
  volume = {43},
  number = {5},
  doi = {10.1093/fampra/cmag071}
}
Kostenlos starten →