Detecting stigmatizing language with large language models: mind the settings
JAMIA Open · 06.03.2026 · 9 (2)
DOI: 10.1093/jamiaopen/ooag037 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 02.04.2026. Verfasst von Teenu Xavier und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Nursing, George Mason University, Fairfax, VA 22030, United States. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Stigmatizing language in clinical documentation can contribute to healthcare disparities and affect patient–provider relationships. Given their strong capacity for contextual language understanding, large language models (LLMs) offer potential for detecting and reducing such language. This study evaluates the accuracy of LLMs in detecting stigmatizing language, focusing on model size, temperature settings, and the inclusion of examples.
Methods
We evaluated multiple configurations of 2 local Llama-based large language models, Llama 3.2 (3B) and Llama 3.1 (8B) with varying temperature (0.25, 0.5, 0.75) and the inclusion of exemple prompts. The models were evaluated on 3643 de-identified clinical notes obtained from a tertiary care teaching hospital. Performance was assessed using accuracy, True Positive Rate (TPR), and True Negative Rate (TNR), with human annotator performance used as a benchmark.
Results
The 8B model with a temperature of 0.25 and examples achieved the highest overall accuracy (70.2%), with the best TPR (94.1%), but the lowest TNR (47.4%). The 3B model without examples achieved the highest TNR (99.7%) but a very low TPR (2%). The inclusion of examples improved model accuracy across all configurations, while temperature settings had a variable impact, with smaller models benefiting from higher temperatures and larger models performing better at lower temperatures. ED provider notes showed higher accuracy (69.4%) and the plan of care was the lowest (55.8%).
Conclusion
Model size, temperature, and the inclusion of examples play a critical role in optimizing open-source LLM performance. Tailoring these parameters to note types enhances effectiveness. Further research should refine these models for broader clinical application and assess their potential to reduce bias in healthcare documentation.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Nursing, George Mason University, Fairfax, VA 22030, United States.
- College of Nursing, University of Florida, FL 32603, United States.
- College of Nursing, University of Cincinnati, Cincinnati, OH 45221, United States.
ORCID-Profile: Teenu Xavier
Zitieren
APA
Xavier, T., Carrington, J. M., & Lambert W, J. (2026). Detecting stigmatizing language with large language models: mind the settings. JAMIA Open, 9(2). https://doi.org/10.1093/jamiaopen/ooag037
Vancouver
Xavier T, Carrington JM, Lambert W J. Detecting stigmatizing language with large language models: mind the settings. JAMIA Open. 2026;9(2). doi:10.1093/jamiaopen/ooag037
BibTeX
@article{Xavier2026,
author = {Xavier, Teenu and Carrington, Jane M and Lambert W, Joshua},
title = {Detecting stigmatizing language with large language models: mind the settings},
journal = {JAMIA Open},
year = {2026},
volume = {9},
number = {2},
doi = {10.1093/jamiaopen/ooag037}
}
EUVIMED