EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Benchmarking Large Language Models for News Summarization

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto

Transactions of the Association for Computational Linguistics · 31.01.2024 · 12 · 39-57

DOI: 10.1162/tacl_a_00632 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 31.01.2024. Verfasst von Tianyi Zhang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 361-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.

Zitieren

APA

Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024). Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12, 39-57. https://doi.org/10.1162/tacl_a_00632

Vancouver

Zhang T, Ladhak F, Durmus E, Liang P, McKeown K, Hashimoto TB. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics. 2024;12:39-57. doi:10.1162/tacl_a_00632

BibTeX
@article{Zhang2024,
  author = {Zhang, Tianyi and Ladhak, Faisal and Durmus, Esin and Liang, Percy and McKeown, Kathleen and Hashimoto, Tatsunori B.},
  title = {Benchmarking Large Language Models for News Summarization},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  volume = {12},
  pages = {39--57},
  doi = {10.1162/tacl_a_00632}
}
Kostenlos starten →