Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation
Language Testing · 2026-10 · 43 (4) · 549-578
DOI: 10.1177/02655322261453702 ↗Kurz gefasst
Fachartikel in der Zeitschrift Language Testing, online erschienen am 28.06.2026. Verfasst von Dana Gablasova und 5 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Economic and Social Research Council.
Abstract (Originalsprache)
This study presents a methodological framework for applying corpus linguistics to systematically evaluate the authenticity of chatbot production in relation to (spoken) production in a general target language use domain. We demonstrate the approach through data drawn from the development cycle of a low-stakes formative assessment system in which learners interact with a ChatGPT-powered bot. A Chatbot Corpus containing approx. 290,000 words from 600 simulations of target ChatGPT production was created, representing two GPT versions (3.5 and 4), and three temperature settings. This corpus was then compared with relevant subcorpora in the British National Corpus 2014, which contains 100 million words of British English collected in naturalistic settings. Analyses were conducted at macro- (multi-dimensional analysis), meso- (comparative frequency analysis), and micro-levels (occurrence of specific pragmatic feature analysis). Results showed that the ChatGPT-powered chatbot production was systematically more similar to genres of written rather than spoken communication: output demonstrated higher lexical density and was characterised by a relatively low occurrence of features typical of spoken communication such as stance and pragmatic markers. We argue that the methodological framework is applicable across different chatbot models, allowing researchers and developers to use this approach with newer, more refined AI-powered conversational agents in the future.
Förderung
- Economic and Social Research Council
Zitieren
APA
Gablasova, D., Harding, L., Brezina, V., Hazelhurst, E. T., O’Sullivan, B., & Spiby, R. (2026). Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation. Language Testing, 43(4), 549-578. https://doi.org/10.1177/02655322261453702
Vancouver
Gablasova D, Harding L, Brezina V, Hazelhurst ET, O’Sullivan B, Spiby R. Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation. Language Testing. 2026;43(4):549-578. doi:10.1177/02655322261453702
BibTeX
@article{Gablasova2026,
author = {Gablasova, Dana and Harding, Luke and Brezina, Vaclav and Hazelhurst, Emil T. and O’Sullivan, Barry and Spiby, Richard},
title = {Evaluating Chatbot Authenticity in Simulations of Spoken Interaction: Demonstrating The Utility of Corpus-Based Methods for Development and Validation},
journal = {Language Testing},
year = {2026},
volume = {43},
number = {4},
pages = {549--578},
doi = {10.1177/02655322261453702}
}
EUVIMED