IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance
Transactions of the Association for Computational Linguistics · 01.04.2026 · 14 · 318-340
DOI: 10.1162/tacl.a.626 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.04.2026. Verfasst von Paul Röttger und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g., “write a blog about”) and 212 political issues (e.g., “AI regulation”) from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.
Zitieren
APA
Röttger, P., Hinck, M., Hofmann, V., Hackenburg, K., Pyatkin, V., Brahman, F., & Hovy, D. (2026). IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance. Transactions of the Association for Computational Linguistics, 14, 318-340. https://doi.org/10.1162/tacl.a.626
Vancouver
Röttger P, Hinck M, Hofmann V, Hackenburg K, Pyatkin V, Brahman F, et al. IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance. Transactions of the Association for Computational Linguistics. 2026;14:318-340. doi:10.1162/tacl.a.626
BibTeX
@article{Rttger2026,
author = {Röttger, Paul and Hinck, Musashi and Hofmann, Valentin and Hackenburg, Kobi and Pyatkin, Valentina and Brahman, Faeze and Hovy, Dirk},
title = {IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {318--340},
doi = {10.1162/tacl.a.626}
}
EUVIMED