When AI Becomes Its Own Biggest Fan: Self-Preference Bias in AI-Assisted Peer Review
IEEE Transactions on Engineering Management · 2026 · 73 · 3710-3722
DOI: 10.1109/tem.2026.3691360 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Engineering Management, 2026. Verfasst von Jianzheng Shi und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben.
Abstract (Originalsprache)
This study examined whether large language models exhibit self-preference bias, a tendency to favor texts generated by themselves over those produced by other models, in artificial intelligence (AI)-assisted peer review. Using a controlled review simulation, GPT-4, GPT-3, and LLaMA were used to generate and evaluate 60 academic manuscripts under a common rubric covering clarity, relevance, originality, and methodological appropriateness. This design enabled direct comparison between self-reviews (a model evaluates its own generated manuscript) and cross-reviews (a model evaluates manuscripts generated by other models) under identical evaluation conditions. In addition, BERT was used to compute embedding-based semantic similarity as a robustness measure. The results indicated that GPT-4 and GPT-3 exhibited strong self-preference bias, assigning significantly higher scores to their own outputs than to manuscripts generated by other models. Human review scores were more closely aligned with AI cross-reviews than AI self-reviews, whereas LLaMA showed weaker and less consistent self-preference patterns, corresponding to its higher rejection rate during quality screening. These findings remained robust across prompt-sensitivity, resampling, alternative aggregation, subsample, and model-specification checks. Taken together, the findings point to a structural source of distortion in AI-assisted evaluation, suggesting that bias may arise not only from training data but also from how generation and evaluation roles are configured within a workflow. The study contributes to research on AI-enabled evaluation systems by extending current discussions of algorithmic bias toward workflow design, and it offers practical guidance on cross-model evaluation, calibration, and human-in-the-loop safeguards in high-stakes evaluative settings.
Zitieren
APA
Shi, J., Lee, J., Xu, L. Z., Ow, T. T., & Wang, Y. (2026). When AI Becomes Its Own Biggest Fan: Self-Preference Bias in AI-Assisted Peer Review. IEEE Transactions on Engineering Management, 73, 3710-3722. https://doi.org/10.1109/tem.2026.3691360
Vancouver
Shi J, Lee J, Xu LZ, Ow TT, Wang Y. When AI Becomes Its Own Biggest Fan: Self-Preference Bias in AI-Assisted Peer Review. IEEE Transactions on Engineering Management. 2026;73:3710-3722. doi:10.1109/tem.2026.3691360
BibTeX
@article{Shi2026,
author = {Shi, Jianzheng and Lee, Jungmin and Xu, Larry Zhiming and Ow, Terence T. and Wang, Yue},
title = {When AI Becomes Its Own Biggest Fan: Self-Preference Bias in AI-Assisted Peer Review},
journal = {IEEE Transactions on Engineering Management},
year = {2026},
volume = {73},
pages = {3710--3722},
doi = {10.1109/tem.2026.3691360}
}
EUVIMED