Source framing triggers systematic bias in large language models
Science Advances · 07.11.2025 · 11 (45)
DOI: 10.1126/sciadv.adz2924 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science Advances, erschienen am 07.11.2025. Verfasst von Federico Germani und 1 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Biomedical Ethics and History of Medicine, University of Zurich, Zurich, Switzerland. Hauptthemen laut MEDLINE: Language, Models, Theoretical. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Large language models (LLMs) are increasingly used to evaluate text, raising urgent questions about whether their judgments are consistent, unbiased, and robust to framing effects. Here, we examine inter- and intramodel agreement across four state-of-the-art LLMs tasked with evaluating 4800 narrative statements on 24 different topics of social, political, and public health relevance, for a total of 192,000 assessments. We manipulate the disclosed source of each statement to assess how attribution to either another LLM or a human author of specified nationality affects evaluation outcomes. Different LLMs display a remarkably high degree of inter- and intramodel agreement across topics, but this alignment breaks down when source framing is introduced. Attributing statements to Chinese individuals systematically lowers agreement scores across all models and, in particular, for DeepSeek Reasoner. Our findings show that LLMs’ own judgment of agreement with narrative statements exhibit systematic bias from framing effects, with substantial implications for the neutrality and fairness of LLM-mediated information systems.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Institute of Biomedical Ethics and History of Medicine, University of Zurich, Zurich, Switzerland.
- Center for Medical Ethics, University of Oslo, Oslo, Norway.
ORCID-Profile: Federico Germani, Giovanni Spitale
Zitieren
APA
Germani, F., & Spitale, G. (2025). Source framing triggers systematic bias in large language models. Science Advances, 11(45). https://doi.org/10.1126/sciadv.adz2924
Vancouver
Germani F, Spitale G. Source framing triggers systematic bias in large language models. Science Advances. 2025;11(45). doi:10.1126/sciadv.adz2924
BibTeX
@article{Germani2025,
author = {Germani, Federico and Spitale, Giovanni},
title = {Source framing triggers systematic bias in large language models},
journal = {Science Advances},
year = {2025},
volume = {11},
number = {45},
doi = {10.1126/sciadv.adz2924}
}
EUVIMED