EUVIMEDEUVIMED entdecken
Crossref

Language Models in the Loop: Incorporating Prompting into Weak Supervision

Ryan Smith, Jason A. Fries, Braden Hancock, Stephen H. Bach

ACM / IMS Journal of Data Science · 30.06.2024 · 1 (2) · 1-30

DOI: 10.1145/3617130 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM / IMS Journal of Data Science, online erschienen am 08.04.2024. Verfasst von Ryan Smith und 3 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 70 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct queries about an example and define how the possible responses should be mapped to votes for labels and abstentions. We then denoise these noisy label sources using the Snorkel system and train an end classifier with the resulting training data. Our experimental evaluation shows that prompting large language models within a weak supervision framework can provide significant gains in accuracy. On the WRENCH weak supervision benchmark, this approach can significantly improve over zero-shot performance, an average 19.5% reduction in errors. We also find that this approach produces classifiers with comparable or superior accuracy to those trained from hand-engineered rules.

Zitieren

APA

Smith, R., Fries, J. A., Hancock, B., & Bach, S. H. (2024). Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM / IMS Journal of Data Science, 1(2), 1-30. https://doi.org/10.1145/3617130

Vancouver

Smith R, Fries JA, Hancock B, Bach SH. Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM / IMS Journal of Data Science. 2024;1(2):1-30. doi:10.1145/3617130

BibTeX
@article{Smith2024,
  author = {Smith, Ryan and Fries, Jason A. and Hancock, Braden and Bach, Stephen H.},
  title = {Language Models in the Loop: Incorporating Prompting into Weak Supervision},
  journal = {ACM / IMS Journal of Data Science},
  year = {2024},
  volume = {1},
  number = {2},
  pages = {1--30},
  doi = {10.1145/3617130}
}
Kostenlos starten →