A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring
International Journal of World Economic Research · 17.11.2026 · 4 (1) · 34-42
DOI: 10.54097/navkme51 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of World Economic Research, online erschienen am 17.11.2026. Verfasst von Leila Diaz und 2 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Large language model (LLM) representations have become the default backbone of automated essay scoring, yet the resulting scores may react to construct-irrelevant features of African American English (AAE) and learner (L2) English. We present a comparative counterfactual study of linguistic bias across five representation families—word and character n-grams, static word vectors, LLM-derived static token embeddings (WordLlama) and RoBERTa examined layer by layer—on 5,569 ASAP essays. Real essays are rewritten with eight AAE morphosyntactic rules and with learner errors whose rates and misspellings are estimated from JFLEG and TOEFL-Spell, and every edit is labelled so that score shifts can be attributed to fourteen phenomena. All unmitigated scorers lower the scores of counterfactual variants, but the mean shifts stay below 0.06 standard deviations while 4.0–9.7% of individual reported scores change, so group-level screening would accept scorers that alter individual outcomes. Within RoBERTa, sensitivity to dialect grows with depth: the AAE penalty increases from −0.015 standard deviations at the embedding output to −0.077 at layer 10, and the contextual model penalises 13 of the 14 phenomena. Construct-aligned invariance regularisation lowers counterfactual flip rates on content scores for every family (by 28–60%) with QWK changes between −0.007 and +0.004. We extend it with variety-selective construct maps and sensitivity anchoring: on the Conventions trait of RoBERTa, the extension removes 80% of the AAE penalty while retaining 98% of the learner-error penalty, whereas a uniform map retains 66%. A geometric bound and the measured overlap of dialect and learner displacement directions show why surface-identical phenomena, such as a missing third-person -s, cannot be treated differently by any linear scorer.
Zitieren
APA
Diaz, L., Sato, L., & Yoon, J. (2026). A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring. International Journal of World Economic Research, 4(1), 34-42. https://doi.org/10.54097/navkme51
Vancouver
Diaz L, Sato L, Yoon J. A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring. International Journal of World Economic Research. 2026;4(1):34-42. doi:10.54097/navkme51
BibTeX
@article{Diaz2026,
author = {Diaz, Leila and Sato, Liam and Yoon, Jasmine},
title = {A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring},
journal = {International Journal of World Economic Research},
year = {2026},
volume = {4},
number = {1},
pages = {34--42},
doi = {10.54097/navkme51}
}
EUVIMED