Language Models in the Loop: Incorporating Prompting into Weak Supervision
ACM / IMS Journal of Data Science · 30.06.2024 · 1 (2) · 1-30
DOI: 10.1145/3617130 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM / IMS Journal of Data Science, online erschienen am 08.04.2024. Verfasst von Ryan Smith und 3 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 70 Literaturangaben.
Abstract (Originalsprache)
We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct queries about an example and define how the possible responses should be mapped to votes for labels and abstentions. We then denoise these noisy label sources using the Snorkel system and train an end classifier with the resulting training data. Our experimental evaluation shows that prompting large language models within a weak supervision framework can provide significant gains in accuracy. On the WRENCH weak supervision benchmark, this approach can significantly improve over zero-shot performance, an average 19.5% reduction in errors. We also find that this approach produces classifiers with comparable or superior accuracy to those trained from hand-engineered rules.
Zitieren
APA
Smith, R., Fries, J. A., Hancock, B., & Bach, S. H. (2024). Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM / IMS Journal of Data Science, 1(2), 1-30. https://doi.org/10.1145/3617130
Vancouver
Smith R, Fries JA, Hancock B, Bach SH. Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM / IMS Journal of Data Science. 2024;1(2):1-30. doi:10.1145/3617130
BibTeX
@article{Smith2024,
author = {Smith, Ryan and Fries, Jason A. and Hancock, Braden and Bach, Stephen H.},
title = {Language Models in the Loop: Incorporating Prompting into Weak Supervision},
journal = {ACM / IMS Journal of Data Science},
year = {2024},
volume = {1},
number = {2},
pages = {1--30},
doi = {10.1145/3617130}
}
EUVIMED