EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation

Dana Gablasova, Luke Harding, Vaclav Brezina, Emil T. Hazelhurst, Barry O’Sullivan, Richard Spiby

Language Testing · 2026-10 · 43 (4) · 549-578

DOI: 10.1177/02655322261453702 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Language Testing, online erschienen am 28.06.2026. Verfasst von Dana Gablasova und 5 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Economic and Social Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study presents a methodological framework for applying corpus linguistics to systematically evaluate the authenticity of chatbot production in relation to (spoken) production in a general target language use domain. We demonstrate the approach through data drawn from the development cycle of a low-stakes formative assessment system in which learners interact with a ChatGPT-powered bot. A Chatbot Corpus containing approx. 290,000 words from 600 simulations of target ChatGPT production was created, representing two GPT versions (3.5 and 4), and three temperature settings. This corpus was then compared with relevant subcorpora in the British National Corpus 2014, which contains 100 million words of British English collected in naturalistic settings. Analyses were conducted at macro- (multi-dimensional analysis), meso- (comparative frequency analysis), and micro-levels (occurrence of specific pragmatic feature analysis). Results showed that the ChatGPT-powered chatbot production was systematically more similar to genres of written rather than spoken communication: output demonstrated higher lexical density and was characterised by a relatively low occurrence of features typical of spoken communication such as stance and pragmatic markers. We argue that the methodological framework is applicable across different chatbot models, allowing researchers and developers to use this approach with newer, more refined AI-powered conversational agents in the future.

Förderung

  • Economic and Social Research Council

Zitieren

APA

Gablasova, D., Harding, L., Brezina, V., Hazelhurst, E. T., O’Sullivan, B., & Spiby, R. (2026). Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation. Language Testing, 43(4), 549-578. https://doi.org/10.1177/02655322261453702

Vancouver

Gablasova D, Harding L, Brezina V, Hazelhurst ET, O’Sullivan B, Spiby R. Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation. Language Testing. 2026;43(4):549-578. doi:10.1177/02655322261453702

BibTeX
@article{Gablasova2026,
  author = {Gablasova, Dana and Harding, Luke and Brezina, Vaclav and Hazelhurst, Emil T. and O’Sullivan, Barry and Spiby, Richard},
  title = {Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation},
  journal = {Language Testing},
  year = {2026},
  volume = {43},
  number = {4},
  pages = {549--578},
  doi = {10.1177/02655322261453702}
}
Kostenlos starten →