Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language
The Modern Language Journal · 20.08.2026
DOI: 10.1111/modl.70089 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Modern Language Journal, online erschienen am 20.08.2026. Verfasst von Peter Crosthwaite und 2 weiteren Autor:innen. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Tianjin Municipal Education Commission.
Abstract (Originalsprache)
Recent corpus‐based studies suggest generative AI systems such as ChatGPT can approximate human patterns of interpersonal metadiscourse in argumentative writing. These conclusions are typically based on AI‐generated texts produced using a single prompt, despite AI output being highly sensitive to even minor changes in prompt specification. This study examines prompt specificity as an elicitation variable in the production of metadiscourse. Using Hyland's interpersonal metadiscourse framework, we first compared university students’ argumentative essays from the LOCNESS corpus with GPT‐generated essays produced in response to a general essay (baseline) prompt specifying role, genre, topic, length and persuasive purpose, but providing no explicit guidance regarding metadiscourse use. We then generated a second GPT corpus using a metadiscourse‐inducing prompt that explicitly requested interactive, interactional, and engagement features that were found to be underrepresented in the initial GPT corpus relative to the LOCNESS corpus. The baseline prompt produced lower frequencies of transitions, hedges, self‐mention, attitude markers and engagement resources than human writing, while the metadiscourse‐inducing prompt substantially increased several target features over the baseline and human conditions. Some categories converged with baseline frequencies, but even where convergence occurred, these were often realised in more formulaic ways. The findings show prompt specificity significantly shapes both the distribution and rhetorical realisation of metadiscourse in AI‐generated writing. More broadly, the results highlight the importance of treating prompt specification as a key methodological variable when interpreting linguistic patterns in AI‐generated texts.
Förderung
- Tianjin Municipal Education Commission
Zitieren
APA
Crosthwaite, P., Sun, S. A., & Jiang, F. K. (2026). Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language. The Modern Language Journal. https://doi.org/10.1111/modl.70089
Vancouver
Crosthwaite P, Sun SA, Jiang FK. Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language. The Modern Language Journal. 2026. doi:10.1111/modl.70089
BibTeX
@article{Crosthwaite2026,
author = {Crosthwaite, Peter and Sun, Shuyi (Amelia) and Jiang, Feng (Kevin)},
title = {Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language},
journal = {The Modern Language Journal},
year = {2026},
doi = {10.1111/modl.70089}
}
EUVIMED