EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text

Dominic Callan, Jennifer Foster

Expert Systems · 2023-07 · 40 (6)

DOI: 10.1111/exsy.13292 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Expert Systems, online erschienen am 27.03.2023. Verfasst von Dominic Callan und 1 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Evaluation of the narrative text generated by machines has traditionally been a challenge, particularly when attempting to evaluate subjective elements such as interest or believability. Recent improvements in narrative machine text generation have been largely driven by the emergence of transformer‐based language models, trained on massive quantities of data, resulting in higher quality text generation. In this study, a corpus of stories is generated using the pre‐trained GPT‐Neo transformer model, with human‐written prompts as inputs upon which to base the narrative text. The stories generated through this process are subsequently evaluated through both human evaluation and two automated metrics: BERTScore and BERT Next Sentence Prediction, with the aim of determining whether there is a correlation between the automatic scores and the human judgements. The results show variation in human evaluation results in comparison to modern automated metrics, suggesting further work is required to train automated metrics to identify text that is defined as interesting by humans.

Zitieren

APA

Callan, D., & Foster, J. (2023). How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text. Expert Systems, 40(6). https://doi.org/10.1111/exsy.13292

Vancouver

Callan D, Foster J. How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text. Expert Systems. 2023;40(6). doi:10.1111/exsy.13292

BibTeX
@article{Callan2023,
  author = {Callan, Dominic and Foster, Jennifer},
  title = {How interesting and coherent are the stories generated by a large‐scale neural language model? Comparing human and automatic evaluations of machine‐generated text},
  journal = {Expert Systems},
  year = {2023},
  volume = {40},
  number = {6},
  doi = {10.1111/exsy.13292}
}
Kostenlos starten →