Evaluating an LLM’s Performance in Annotating Discourse Strategies
Corpus Pragmatics · 2026-06 · 10 (1)
DOI: 10.1007/s41701-025-00224-2 ↗Kurz gefasst
Fachartikel in der Zeitschrift Corpus Pragmatics, online erschienen am 07.01.2026. Verfasst von Taylor Meizlish und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben.
Abstract (Originalsprache)
Manual annotation remains essential for identifying complex pragmatic and discourse-level features in corpus linguistics, particularly the functional components of speech acts. While part-of-speech and semantic tagging can be automated with high accuracy, annotating discourse strategies remains challenging because strategies can appear as a wide range of surface forms that cannot be fully captured by predefined example sets. These limitations hinder the scalability of function-to-form approaches and constrain the development of richly annotated corpora for pragmatics research and instruction. This study investigates whether a large language model (LLM), GPT-4 (accessed via the ChatGPT interface), can assist with the annotation of refusal strategies as they are explicitly realized in written learner responses, rather than inferred or context-dependent meanings. The focus is on identifying clear surface forms that express pragmatic functions. A corpus of written Discourse Completion Tasks completed by Japanese university English learners was analyzed for reliability, GPT-4–human agreement, accuracy, and the model’s ability to generalize beyond the specific examples provided in the prompts. The results suggest that GPT-4 can effectively support this type of form-guided, functional pragmatic annotation and thereby increase scalability and accuracy.
Zitieren
APA
Meizlish, T., & Ziffo, C. (2026). Evaluating an LLM’s Performance in Annotating Discourse Strategies. Corpus Pragmatics, 10(1). https://doi.org/10.1007/s41701-025-00224-2
Vancouver
Meizlish T, Ziffo C. Evaluating an LLM’s Performance in Annotating Discourse Strategies. Corpus Pragmatics. 2026;10(1). doi:10.1007/s41701-025-00224-2
BibTeX
@article{Meizlish2026,
author = {Meizlish, Taylor and Ziffo, Chris},
title = {Evaluating an LLM’s Performance in Annotating Discourse Strategies},
journal = {Corpus Pragmatics},
year = {2026},
volume = {10},
number = {1},
doi = {10.1007/s41701-025-00224-2}
}
EUVIMED