EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Detecting stigmatizing language with large language models: mind the settings

Teenu Xavier iD, Jane M Carrington, Joshua Lambert W

JAMIA Open · 06.03.2026 · 9 (2)

DOI: 10.1093/jamiaopen/ooag037 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 02.04.2026. Verfasst von Teenu Xavier und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Nursing, George Mason University, Fairfax, VA 22030, United States. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Stigmatizing language in clinical documentation can contribute to healthcare disparities and affect patient–provider relationships. Given their strong capacity for contextual language understanding, large language models (LLMs) offer potential for detecting and reducing such language. This study evaluates the accuracy of LLMs in detecting stigmatizing language, focusing on model size, temperature settings, and the inclusion of examples.

Methods

We evaluated multiple configurations of 2 local Llama-based large language models, Llama 3.2 (3B) and Llama 3.1 (8B) with varying temperature (0.25, 0.5, 0.75) and the inclusion of exemple prompts. The models were evaluated on 3643 de-identified clinical notes obtained from a tertiary care teaching hospital. Performance was assessed using accuracy, True Positive Rate (TPR), and True Negative Rate (TNR), with human annotator performance used as a benchmark.

Results

The 8B model with a temperature of 0.25 and examples achieved the highest overall accuracy (70.2%), with the best TPR (94.1%), but the lowest TNR (47.4%). The 3B model without examples achieved the highest TNR (99.7%) but a very low TPR (2%). The inclusion of examples improved model accuracy across all configurations, while temperature settings had a variable impact, with smaller models benefiting from higher temperatures and larger models performing better at lower temperatures. ED provider notes showed higher accuracy (69.4%) and the plan of care was the lowest (55.8%).

Conclusion

Model size, temperature, and the inclusion of examples play a critical role in optimizing open-source LLM performance. Tailoring these parameters to note types enhances effectiveness. Further research should refine these models for broader clinical application and assess their potential to reduce bias in healthcare documentation.

Schlagwörter

Stichwörter der Autor:innen

clinical documentationclinical noteslarge language modelsnatural language processingstigmatizing language

Autor:innen und Institutionen

  • School of Nursing, George Mason University, Fairfax, VA 22030, United States.
  • College of Nursing, University of Florida, FL 32603, United States.
  • College of Nursing, University of Cincinnati, Cincinnati, OH 45221, United States.

ORCID-Profile: Teenu Xavier

Zitieren

APA

Xavier, T., Carrington, J. M., & Lambert W, J. (2026). Detecting stigmatizing language with large language models: mind the settings. JAMIA Open, 9(2). https://doi.org/10.1093/jamiaopen/ooag037

Vancouver

Xavier T, Carrington JM, Lambert W J. Detecting stigmatizing language with large language models: mind the settings. JAMIA Open. 2026;9(2). doi:10.1093/jamiaopen/ooag037

BibTeX
@article{Xavier2026,
  author = {Xavier, Teenu and Carrington, Jane M and Lambert W, Joshua},
  title = {Detecting stigmatizing language with large language models: mind the settings},
  journal = {JAMIA Open},
  year = {2026},
  volume = {9},
  number = {2},
  doi = {10.1093/jamiaopen/ooag037}
}
Kostenlos starten →