EUVIMEDEUVIMED entdecken
Open Access

Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study

Deija McCalla, Bochen Li, Saul Jaeger, Justin Jacques, Leo Gonzalez, Charles Silber, Cass Dykeman

JMIR AI · 23.09.2026 · 5 · e93498-e93498

DOI: 10.2196/93498 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 23.09.2026. Verfasst von Deija McCalla und 6 weiteren Autor:innen. Beteiligte Einrichtung: New Generation Mental Health Counseling, 223 Bedford Ave, Brooklyn, NY, 11211, United States, 1 347-559-7120. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly deployed in mental health applications, yet growing evidence suggests they encode algorithmic biases that influence clinical outputs. Because these models now mediate patient-facing decisions, such biases carry the potential for direct harm. Whether they systematically affect psychiatric diagnosis across demographic groups remains underexplored.

Objective

This study aims to examine whether LLMs exhibit implicit demographic biases when generating psychiatric diagnoses.

Methods

We developed 1152 synthetic clinical vignettes using a matched-pair design that manipulated gender, race and ethnicity, age, socioeconomic status, English proficiency, and urbanicity while holding clinical content constant. Vignettes were divided into control (unambiguous anorexia nervosa [AN]) and ambiguous conditions designed to permit differential diagnosis. Ten LLM configurations across 5 model families were tested.

Results

Control vignettes produced near-unanimous AN diagnoses (mean 100%, SD 0.1%), while ambiguous vignettes elicited greater variability (mean 23.6%, SD 10.1%). Intermodel agreement was moderate for ambiguous vignettes (Fleiss κ=0.410, 95% CI 0.397‐0.422). Mixed-effects logistic regression with LLM as a random intercept revealed significant demographic biases: Black patients were over 6 times more likely to receive a major depressive disorder (MDD) diagnosis than White patients with identical presentations (odds ratio [OR] 6.09, 95% CI 5.13‐7.24), Latine patients were over 9 times more likely (OR 9.57, 95% CI 8.00‐11.45), and Asian patients were nearly 3 times more likely to receive an AN diagnosis (OR 2.88, 95% CI 2.44‐3.42). Female patients were less likely than males to be diagnosed with AN (OR 0.43, 95% CI 0.37‐0.49).

Conclusions

These findings demonstrate that LLMs exhibit systematic demographic biases in psychiatric diagnosis even when clinical content is held constant, revealing measurable patterns that can inform improvements to training data, model architecture, and clinical deployment frameworks.

Schlagwörter

Stichwörter der Autor:innen

algorithmic biasartificial intelligenceeating disordershealth equitylarge language modelspsychiatric diagnosis

Autor:innen und Institutionen

  • New Generation Mental Health Counseling, 223 Bedford Ave, Brooklyn, NY, 11211, United States, 1 347-559-7120.
  • Oregon State University, Corvallis, OR, United States.
  • Touro University, Los Alamitos, CA, United States.
  • Human Theory Group, Washington, DC, United States.
  • St. John's University, Queens, NY, United States.
  • Rutgers University, New Brunswick, NJ, United States.

Zitieren

APA

McCalla, D., Li, B., Jaeger, S., Jacques, J., Gonzalez, L., Silber, C., & Dykeman, C. (2026). Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study. JMIR AI, 5, e93498-e93498. https://doi.org/10.2196/93498

Vancouver

McCalla D, Li B, Jaeger S, Jacques J, Gonzalez L, Silber C, et al. Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study. JMIR AI. 2026;5:e93498-e93498. doi:10.2196/93498

BibTeX
@article{McCalla2026,
  author = {McCalla, Deija and Li, Bochen and Jaeger, Saul and Jacques, Justin and Gonzalez, Leo and Silber, Charles and Dykeman, Cass},
  title = {Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study},
  journal = {JMIR AI},
  year = {2026},
  volume = {5},
  pages = {e93498--e93498},
  doi = {10.2196/93498}
}
Kostenlos starten →