Echoes in AI: Quantifying lack of plot diversity in LLM outputs
Proceedings of the National Academy of Sciences · 02.09.2025 · 122 (35)
DOI: 10.1073/pnas.2504966122 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 28.08.2025. Verfasst von Weijia Xu und 4 weiteren Autor:innen. Beteiligte Einrichtung: Microsoft Research, Redmond, WA 98052. Hauptthemen laut MEDLINE: Language, Artificial Intelligence, Creativity. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
With rapid advances in large language models (LLMs), there has been an increasing application of LLMs in creative content ideation and generation. A critical question emerges: can current LLMs provide ideas that are diverse enough to truly bolster collective creativity? We examine two state-of-the-art LLMs, GPT-4 and LLaMA-3, on story generation and discover that LLM-generated stories often consist of plot elements that are echoed across a number of generations. To quantify this phenomenon, we introduce the Sui Generis score, an automatic metric that measures the uniqueness of a plot element among alternative storylines generated using the same prompt under an LLM. Evaluating on 100 short stories, we find that LLM-generated stories often contain combinations of idiosyncratic plot elements echoed frequently across generations and across different LLMs, while plots from the original human-written stories are rarely recreated or even echoed in pieces. Moreover, our human evaluation shows that the ranking of Sui Generis scores among story segments correlates moderately with human judgment of surprise level, even though score computation is completely automatic without relying on human judgment.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Microsoft Research, Redmond, WA 98052.
ORCID-Profile: Weijia Xu, Nebojsa Jojic
Zitieren
APA
Xu, W., Jojic, N., Rao, S., Brockett, C., & Dolan, B. (2025). Echoes in AI: Quantifying lack of plot diversity in LLM outputs. Proceedings of the National Academy of Sciences, 122(35). https://doi.org/10.1073/pnas.2504966122
Vancouver
Xu W, Jojic N, Rao S, Brockett C, Dolan B. Echoes in AI: Quantifying lack of plot diversity in LLM outputs. Proceedings of the National Academy of Sciences. 2025;122(35). doi:10.1073/pnas.2504966122
BibTeX
@article{Xu2025,
author = {Xu, Weijia and Jojic, Nebojsa and Rao, Sudha and Brockett, Chris and Dolan, Bill},
title = {Echoes in AI: Quantifying lack of plot diversity in LLM outputs},
journal = {Proceedings of the National Academy of Sciences},
year = {2025},
volume = {122},
number = {35},
doi = {10.1073/pnas.2504966122}
}
EUVIMED