EUVIMEDEUVIMED entdecken
Crossref

Evaluating an LLM’s Performance in Annotating Discourse Strategies

Taylor Meizlish, Chris Ziffo

Corpus Pragmatics · 2026-06 · 10 (1)

DOI: 10.1007/s41701-025-00224-2 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Corpus Pragmatics, online erschienen am 07.01.2026. Verfasst von Taylor Meizlish und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Manual annotation remains essential for identifying complex pragmatic and discourse-level features in corpus linguistics, particularly the functional components of speech acts. While part-of-speech and semantic tagging can be automated with high accuracy, annotating discourse strategies remains challenging because strategies can appear as a wide range of surface forms that cannot be fully captured by predefined example sets. These limitations hinder the scalability of function-to-form approaches and constrain the development of richly annotated corpora for pragmatics research and instruction. This study investigates whether a large language model (LLM), GPT-4 (accessed via the ChatGPT interface), can assist with the annotation of refusal strategies as they are explicitly realized in written learner responses, rather than inferred or context-dependent meanings. The focus is on identifying clear surface forms that express pragmatic functions. A corpus of written Discourse Completion Tasks completed by Japanese university English learners was analyzed for reliability, GPT-4–human agreement, accuracy, and the model’s ability to generalize beyond the specific examples provided in the prompts. The results suggest that GPT-4 can effectively support this type of form-guided, functional pragmatic annotation and thereby increase scalability and accuracy.

Zitieren

APA

Meizlish, T., & Ziffo, C. (2026). Evaluating an LLM’s Performance in Annotating Discourse Strategies. Corpus Pragmatics, 10(1). https://doi.org/10.1007/s41701-025-00224-2

Vancouver

Meizlish T, Ziffo C. Evaluating an LLM’s Performance in Annotating Discourse Strategies. Corpus Pragmatics. 2026;10(1). doi:10.1007/s41701-025-00224-2

BibTeX
@article{Meizlish2026,
  author = {Meizlish, Taylor and Ziffo, Chris},
  title = {Evaluating an LLM’s Performance in Annotating Discourse Strategies},
  journal = {Corpus Pragmatics},
  year = {2026},
  volume = {10},
  number = {1},
  doi = {10.1007/s41701-025-00224-2}
}
Kostenlos starten →