Measuring Attribution in Natural Language Generation Models
Computational Linguistics · 01.12.2023 · 49 (4) · 777-840
DOI: 10.1162/coli_a_00486 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computational Linguistics, online erschienen am 01.12.2023. Verfasst von Hannah Rashkin und 9 weiteren Autor:innen. Die Arbeit wurde bisher 78-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.
Zitieren
APA
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023). Measuring Attribution in Natural Language Generation Models. Computational Linguistics, 49(4), 777-840. https://doi.org/10.1162/coli_a_00486
Vancouver
Rashkin H, Nikolaev V, Lamm M, Aroyo L, Collins M, Das D, et al. Measuring Attribution in Natural Language Generation Models. Computational Linguistics. 2023;49(4):777-840. doi:10.1162/coli_a_00486
BibTeX
@article{Rashkin2023,
author = {Rashkin, Hannah and Nikolaev, Vitaly and Lamm, Matthew and Aroyo, Lora and Collins, Michael and Das, Dipanjan and Petrov, Slav and Tomar, Gaurav Singh and Turc, Iulia and Reitter, David},
title = {Measuring Attribution in Natural Language Generation Models},
journal = {Computational Linguistics},
year = {2023},
volume = {49},
number = {4},
pages = {777--840},
doi = {10.1162/coli_a_00486}
}
EUVIMED