Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study
JMIR Medical Informatics · 2025-08-18 · 13 · e68955
DOI: 10.2196/68955 ↗At a glance
Vergleichsstudie in JMIR Medical Informatics, published online on 2025-08-18. Written by Hongbo Chen and 2 co-authors. Institution: Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada. Major MEDLINE topics: Electronic Health Records, Machine Learning, Language, Social Stigma. Cited 10 times so far (Crossref); 107 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The presence of stigmatizing language within electronic health records (EHRs) poses significant risks to patient care by perpetuating biases. While numerous studies have explored the use of supervised machine learning models to detect stigmatizing language automatically, these models require large, annotated datasets, which may not always be readily available. In-context learning (ICL) has emerged as a data-efficient alternative, allowing large language models to adapt to tasks using only instructions and examples.
Objective
We aimed to investigate the efficacy of ICL in detecting stigmatizing language within EHRs under data-scarce conditions.
Methods
We analyzed 5043 sentences from the Medical Information Mart for Intensive Care–IV dataset, which contains EHRs from patients admitted to the emergency department at the Beth Israel Deaconess Medical Center. We compared ICL with zero-shot (textual entailment), few-shot (SetFit), and supervised fine-tuning approaches. The ICL approach used 4 prompting strategies: generic, chain of thought, clue and reasoning prompting, and a newly introduced stigma detection guided prompt. Model fairness was evaluated using the equal performance criterion, measuring true positive rate, false positive rate, and F1-score disparities across protected attributes, including sex, age, and race.
Results
In the zero-shot setting, the best-performing ICL model, GEMMA-2, achieved a mean F1-score of 0.858 (95% CI 0.854-0.862), showing an 18.7% improvement over the best textual entailment model, DEBERTA-M (mean F1-score 0.723, 95% CI 0.718-0.728; P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada.
ORCID profiles: Hongbo Chen, Myrtede Alfred, Eldan Cohen
Cite
APA
Chen, H., Alfred, M., & Cohen, E. (2025). Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study. JMIR Medical Informatics, 13, e68955. https://doi.org/10.2196/68955
Vancouver
Chen H, Alfred M, Cohen E. Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study. JMIR Medical Informatics. 2025;13:e68955. doi:10.2196/68955
BibTeX
@article{Chen2025,
author = {Chen, Hongbo and Alfred, Myrtede and Cohen, Eldan},
title = {Efficient Detection of Stigmatizing Language in Electronic Health Records via In-Context Learning: Comparative Analysis and Validation Study},
journal = {JMIR Medical Informatics},
year = {2025},
volume = {13},
pages = {e68955},
doi = {10.2196/68955}
}
EUVIMED