EUVIMEDDescoperă EUVIMED
VergleichsstudieValidierungsstudieOpen access

Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study

Hongbo Chen iD, Myrtede Alfred iD, Eldan Cohen iD

JMIR Medical Informatics · 2025-08-18 · 13 · e68955

DOI: 10.2196/68955 ↗

At a glance

Vergleichsstudie in JMIR Medical Informatics, published online on 2025-08-18. Written by Hongbo Chen and 2 co-authors. Institution: Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada. Major MEDLINE topics: Electronic Health Records, Machine Learning, Language, Social Stigma. Cited 10 times so far (Crossref); 107 references. The full text is freely available in PubMed Central.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Background

The presence of stigmatizing language within electronic health records (EHRs) poses significant risks to patient care by perpetuating biases. While numerous studies have explored the use of supervised machine learning models to detect stigmatizing language automatically, these models require large, annotated datasets, which may not always be readily available. In-context learning (ICL) has emerged as a data-efficient alternative, allowing large language models to adapt to tasks using only instructions and examples.

Objective

We aimed to investigate the efficacy of ICL in detecting stigmatizing language within EHRs under data-scarce conditions.

Methods

We analyzed 5043 sentences from the Medical Information Mart for Intensive Care–IV dataset, which contains EHRs from patients admitted to the emergency department at the Beth Israel Deaconess Medical Center. We compared ICL with zero-shot (textual entailment), few-shot (SetFit), and supervised fine-tuning approaches. The ICL approach used 4 prompting strategies: generic, chain of thought, clue and reasoning prompting, and a newly introduced stigma detection guided prompt. Model fairness was evaluated using the equal performance criterion, measuring true positive rate, false positive rate, and F1-score disparities across protected attributes, including sex, age, and race.

Results

In the zero-shot setting, the best-performing ICL model, GEMMA-2, achieved a mean F1-score of 0.858 (95% CI 0.854-0.862), showing an 18.7% improvement over the best textual entailment model, DEBERTA-M (mean F1-score 0.723, 95% CI 0.718-0.728; P

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsMachine LearningLanguageSocial StigmaStereotypingHumansMaleFemale

Author keywords

artificial intelligenceelectronic health recordfairnessfew-shotin-context learninglarge language modelmachine learningprompting strategystigmatizing languagetext classificationzero-shot

Authors and institutions

  • Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada.

ORCID profiles: Hongbo Chen, Myrtede Alfred, Eldan Cohen

Cite

APA

Chen, H., Alfred, M., & Cohen, E. (2025). Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study. JMIR Medical Informatics, 13, e68955. https://doi.org/10.2196/68955

Vancouver

Chen H, Alfred M, Cohen E. Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study. JMIR Medical Informatics. 2025;13:e68955. doi:10.2196/68955

BibTeX
@article{Chen2025,
  author = {Chen, Hongbo and Alfred, Myrtede and Cohen, Eldan},
  title = {Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study},
  journal = {JMIR Medical Informatics},
  year = {2025},
  volume = {13},
  pages = {e68955},
  doi = {10.2196/68955}
}
Start for free →