EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Much Ado about Prompting: LLM classification of text messages from experiments

Can Çelebi iD, Stefan P Penczynski

PLOS One · 04.08.2026 · 21 (8) · e0354757

DOI: 10.1371/journal.pone.0354757 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 04.08.2026. Verfasst von Can Çelebi und 1 weiteren Autor:innen. Beteiligte Einrichtung: Vienna Center for Experimental Economics (VCEE), University of Vienna, Oskar-Morgenstern-Platz 1, Vienna, Austria. Hauptthemen laut MEDLINE: Text Messaging. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Austrian Science Fund.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Researchers who classify text with large language models often possess codebooks written for human annotators. We ask how these codebooks can serve as prompts. Using three codebooks from experimental economics (one on promise classification, two on strategic thinking), we vary the level of information in the prompt, then vary its formatting, framing and wording at the lowest and highest information levels, across two proprietary and two open-weight models. Used as prompts, the codebooks reach 82–88% agreement with human annotators across the three tasks. On the recognition-heavy task (promise classification), model choice accounts for most of the variation in accuracy; on the learning-heavy tasks (strategic thinking classification), the level of detail in the classification instructions carries comparable weight. These information components partly substitute for one another, whereas model reasoning does not reliably compensate for missing content and yields little or no improvement once the content is present. Larger models make better use of additional information and are more robust to formatting, framing, and wording of the prompt, while smaller models can be hurt by extra information and are more sensitive to how the information is presented. Our results advocate for a shift in focus from prompt engineering techniques (formatting, framing, reasoning, etc.) to the content of the prompt: preparing instructions as one would for human annotators, with detailed context, category definitions, and examples.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Text MessagingLarge Language ModelsHumansClassification Algorithms

Autor:innen und Institutionen

  • Vienna Center for Experimental Economics (VCEE), University of Vienna, Oskar-Morgenstern-Platz 1, Vienna, Austria.
  • School of Economics and Centre for Behavioural and Experimental Social Science (CBESS), University of East Anglia, Norwich Research Park, Norwich, United Kingdom.

ORCID-Profile: Can Çelebi

Förderung

  • Austrian Science Fund

Zitieren

APA

Çelebi, C., & Penczynski, S. P. (2026). Much Ado about Prompting: LLM classification of text messages from experiments. PLOS One, 21(8), e0354757. https://doi.org/10.1371/journal.pone.0354757

Vancouver

Çelebi C, Penczynski SP. Much Ado about Prompting: LLM classification of text messages from experiments. PLOS One. 2026;21(8):e0354757. doi:10.1371/journal.pone.0354757

BibTeX
@article{elebi2026,
  author = {Çelebi, Can and Penczynski, Stefan P},
  title = {Much Ado about Prompting: LLM classification of text messages from experiments},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {8},
  pages = {e0354757},
  doi = {10.1371/journal.pone.0354757}
}
Kostenlos starten →