EUVIMEDEUVIMED entdecken
Crossref

Using Keystroke Behavior Patterns to Detect Nonauthentic Texts in Writing Assessments: Evaluating the Fairness of Predictive Models

Yang Jiang, Mo Zhang, Jiangang Hao, Paul Deane, Chen Li

Journal of Educational Measurement · 2024-12 · 61 (4) · 571-594

DOI: 10.1111/jedm.12416 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Educational Measurement, online erschienen am 18.10.2024. Verfasst von Yang Jiang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 48 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The emergence of sophisticated AI tools such as ChatGPT, coupled with the transition to remote delivery of educational assessments in the COVID‐19 era, has led to increasing concerns about academic integrity and test security. Using AI tools, test takers can produce high‐quality texts effortlessly and use them to game assessments. It is thus critical to detect these nonauthentic texts to ensure test integrity. In this study, we leveraged keystroke logs—recordings of every keypress—to build machine learning (ML) detectors of nonauthentic texts in a large‐scale writing assessment. We focused on investigating the fairness of the detectors across demographic subgroups to ensure that nongenuine writing can be predicted equally well across subgroups. Results indicated that keystroke dynamics were effective in identifying nonauthentic texts. While the ML models were slightly more likely to misclassify the original responses submitted by male test takers as consisting of nonauthentic texts than those submitted by females, the effect sizes were negligible. Furthermore, balancing demographic distributions and class labels did not consistently mitigate detector bias across predictive models. Findings of this study not only provide implications for using behavioral data to address test security issues, but also highlight the importance of evaluating the fairness of predictive models in educational contexts.

Zitieren

APA

Jiang, Y., Zhang, M., Hao, J., Deane, P., & Li, C. (2024). Using Keystroke Behavior Patterns to Detect Nonauthentic Texts in Writing Assessments: Evaluating the Fairness of Predictive Models. Journal of Educational Measurement, 61(4), 571-594. https://doi.org/10.1111/jedm.12416

Vancouver

Jiang Y, Zhang M, Hao J, Deane P, Li C. Using Keystroke Behavior Patterns to Detect Nonauthentic Texts in Writing Assessments: Evaluating the Fairness of Predictive Models. Journal of Educational Measurement. 2024;61(4):571-594. doi:10.1111/jedm.12416

BibTeX
@article{Jiang2024,
  author = {Jiang, Yang and Zhang, Mo and Hao, Jiangang and Deane, Paul and Li, Chen},
  title = {Using Keystroke Behavior Patterns to Detect Nonauthentic Texts in Writing Assessments: Evaluating the Fairness of Predictive Models},
  journal = {Journal of Educational Measurement},
  year = {2024},
  volume = {61},
  number = {4},
  pages = {571--594},
  doi = {10.1111/jedm.12416}
}
Kostenlos starten →