Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
Applied Sciences · 10.03.2025 · 15 (6) · 2971
DOI: 10.3390/app15062971 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 10.03.2025. Verfasst von Sungeun Kim und 1 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Kyungpook National University Research Fund.
Abstract (Originalsprache)
The evaluation of creative writing has long been a complex and subjective process, made even more intriguing by the rise of advanced Artificial Intelligence (AI) tools like Large Language Models (LLMs). This study evaluates the potential of LLMs as reliable and consistent evaluators of creative texts, directly comparing their performance with traditional human evaluations. The analysis focuses on key creative criteria, including fluency, flexibility, elaboration, originality, usefulness, and specific creativity strategies. Results demonstrate that LLMs provide consistent and objective evaluations, achieving higher Inter-Annotator Agreement (IAA) compared with human evaluators. However, LLMs face limitations in recognizing nuanced, culturally specific, and context-dependent aspects of creativity. Conversely, human evaluators, despite lower consistency and higher subjectivity, exhibit strengths in capturing deeper contextual insights. These findings highlight the need for the further refinement of LLMs to address the complexities of creative writing evaluation.
Förderung
- Kyungpook National University Research Fund
Zitieren
APA
Kim, S., & Oh, D. (2025). Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?. Applied Sciences, 15(6), 2971. https://doi.org/10.3390/app15062971
Vancouver
Kim S, Oh D. Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?. Applied Sciences. 2025;15(6):2971. doi:10.3390/app15062971
BibTeX
@article{Kim2025,
author = {Kim, Sungeun and Oh, Dongsuk},
title = {Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {6},
pages = {2971},
doi = {10.3390/app15062971}
}
EUVIMED