EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

BIAS-AWARE LARGE LANGUAGE MODELS FOR EQUITABLE EDUCATIONAL FEEDBACK AND ASSESSMENT

QianYi Fang

Trends in Social Sciences and Humanities Research · 02.02.2024 · 2 (1) · 41-50

DOI: 10.61784/tsshr4253 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Trends in Social Sciences and Humanities Research, online erschienen am 02.02.2024. Verfasst von QianYi Fang. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Pre-trained large language models (LLMs) are increasingly used to score student writing and to produce trait-level feedback, yet their sensitivity to construct-irrelevant linguistic variation can disadvantage students who write in African American English (AAE) or who are learning English as a second language (L2). This paper presents a bias-aware scoring framework that audits and mitigates such variation without access to student demographics. Counterfactual versions of real student essays are generated by an AAE morphosyntactic rule system and by an L2 error generator whose error-category rates are estimated from the JFLEG learner corpus and whose misspellings are drawn from TOEFL-Spell. Paired representations from a frozen RoBERTa encoder define an invariance operator, and Construct-Aligned Invariance Regularization (CAIR) applies it only to score dimensions that the rubric defines as independent of language conventions, which yields a closed-form estimator. On 5,569 ASAP essays from five prompts and five random splits, CAIR reduces the mean standardized counterfactual shift of holistic content scores by 85% and the counterfactual score-flip rate by about 40%, with no significant change in quadratic weighted kappa (0.732 vs. 0.733). For trait feedback, CAIR removes the spill-over of learner errors into Ideas, Organization and Style scores while preserving the legitimate sensitivity of the Conventions score, which task-agnostic debiasing methods erase. Pairs for 2% of the training essays already halve the shift, and invariance learned from one variety transfers to the other.

Zitieren

APA

Fang, Q. (2024). BIAS-AWARE LARGE LANGUAGE MODELS FOR EQUITABLE EDUCATIONAL FEEDBACK AND ASSESSMENT. Trends in Social Sciences and Humanities Research, 2(1), 41-50. https://doi.org/10.61784/tsshr4253

Vancouver

Fang Q. BIAS-AWARE LARGE LANGUAGE MODELS FOR EQUITABLE EDUCATIONAL FEEDBACK AND ASSESSMENT. Trends in Social Sciences and Humanities Research. 2024;2(1):41-50. doi:10.61784/tsshr4253

BibTeX
@article{Fang2024,
  author = {Fang, QianYi},
  title = {BIAS-AWARE LARGE LANGUAGE MODELS FOR EQUITABLE EDUCATIONAL FEEDBACK AND ASSESSMENT},
  journal = {Trends in Social Sciences and Humanities Research},
  year = {2024},
  volume = {2},
  number = {1},
  pages = {41--50},
  doi = {10.61784/tsshr4253}
}
Kostenlos starten →