How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text
Expert Systems · 2023-07 · 40 (6)
DOI: 10.1111/exsy.13292 ↗Kurz gefasst
Fachartikel in der Zeitschrift Expert Systems, online erschienen am 27.03.2023. Verfasst von Dominic Callan und 1 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Evaluation of the narrative text generated by machines has traditionally been a challenge, particularly when attempting to evaluate subjective elements such as interest or believability. Recent improvements in narrative machine text generation have been largely driven by the emergence of transformer‐based language models, trained on massive quantities of data, resulting in higher quality text generation. In this study, a corpus of stories is generated using the pre‐trained GPT‐Neo transformer model, with human‐written prompts as inputs upon which to base the narrative text. The stories generated through this process are subsequently evaluated through both human evaluation and two automated metrics: BERTScore and BERT Next Sentence Prediction, with the aim of determining whether there is a correlation between the automatic scores and the human judgements. The results show variation in human evaluation results in comparison to modern automated metrics, suggesting further work is required to train automated metrics to identify text that is defined as interesting by humans.
Zitieren
APA
Callan, D., & Foster, J. (2023). How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text. Expert Systems, 40(6). https://doi.org/10.1111/exsy.13292
Vancouver
Callan D, Foster J. How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text. Expert Systems. 2023;40(6). doi:10.1111/exsy.13292
BibTeX
@article{Callan2023,
author = {Callan, Dominic and Foster, Jennifer},
title = {How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text},
journal = {Expert Systems},
year = {2023},
volume = {40},
number = {6},
doi = {10.1111/exsy.13292}
}
EUVIMED