EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring

Leila Diaz, Liam Sato, Jasmine Yoon

International Journal of World Economic Research · 17.11.2026 · 4 (1) · 34-42

DOI: 10.54097/navkme51 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of World Economic Research, online erschienen am 17.11.2026. Verfasst von Leila Diaz und 2 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language model (LLM) representations have become the default backbone of automated essay scoring, yet the resulting scores may react to construct-irrelevant features of African American English (AAE) and learner (L2) English. We present a comparative counterfactual study of linguistic bias across five representation families—word and character n-grams, static word vectors, LLM-derived static token embeddings (WordLlama) and RoBERTa examined layer by layer—on 5,569 ASAP essays. Real essays are rewritten with eight AAE morphosyntactic rules and with learner errors whose rates and misspellings are estimated from JFLEG and TOEFL-Spell, and every edit is labelled so that score shifts can be attributed to fourteen phenomena. All unmitigated scorers lower the scores of counterfactual variants, but the mean shifts stay below 0.06 standard deviations while 4.0–9.7% of individual reported scores change, so group-level screening would accept scorers that alter individual outcomes. Within RoBERTa, sensitivity to dialect grows with depth: the AAE penalty increases from −0.015 standard deviations at the embedding output to −0.077 at layer 10, and the contextual model penalises 13 of the 14 phenomena. Construct-aligned invariance regularisation lowers counterfactual flip rates on content scores for every family (by 28–60%) with QWK changes between −0.007 and +0.004. We extend it with variety-selective construct maps and sensitivity anchoring: on the Conventions trait of RoBERTa, the extension removes 80% of the AAE penalty while retaining 98% of the learner-error penalty, whereas a uniform map retains 66%. A geometric bound and the measured overlap of dialect and learner displacement directions show why surface-identical phenomena, such as a missing third-person -s, cannot be treated differently by any linear scorer.

Zitieren

APA

Diaz, L., Sato, L., & Yoon, J. (2026). A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring. International Journal of World Economic Research, 4(1), 34-42. https://doi.org/10.54097/navkme51

Vancouver

Diaz L, Sato L, Yoon J. A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring. International Journal of World Economic Research. 2026;4(1):34-42. doi:10.54097/navkme51

BibTeX
@article{Diaz2026,
  author = {Diaz, Leila and Sato, Liam and Yoon, Jasmine},
  title = {A Comparative Study of Linguistic Bias and Fairness in Large Language Models for Automated Essay Scoring},
  journal = {International Journal of World Economic Research},
  year = {2026},
  volume = {4},
  number = {1},
  pages = {34--42},
  doi = {10.54097/navkme51}
}
Kostenlos starten →