EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm

Matthew C. Myers, Joshua Wilson

International Journal of Artificial Intelligence in Education · 2023-09 · 33 (3) · 609-634

DOI: 10.1007/s40593-022-00301-6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Artificial Intelligence in Education, 2023-09. Verfasst von Matthew C. Myers und 1 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study evaluated the construct validity of six scoring traits of an automated writing evaluation (AWE) system called MI Write . Persuasive essays ( N = 100) written by students in grades 7 and 8 were randomized at the sentence-level using a script written with Python’s NLTK module. Each persuasive essay was randomized 30 times ( n = 3000 total randomizations), and the mean trait scores for each set of randomized iterations were compared to those of the control text across all traits. We were specifically interested in evaluating the effects of randomization on the high-level traits of idea development and organization . Given the rubrics and qualitative feedback provided by MI Write, we hypothesized that these high-level traits ought to be sensitive to sentence-level randomization (i.e., scores should decrease). Overall, complete randomizations did not consistently significantly impact trait scoring for these high-level writing traits. In fact, more than a third of the essays saw significant increases in one or both high-level traits despite randomization, indicating a disconnect between MI Write’s formative feedback and its underlying constructs. Findings have implications for consumers and developers of AWE.

Zitieren

APA

Myers, M. C., & Wilson, J. (2023). Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm. International Journal of Artificial Intelligence in Education, 33(3), 609-634. https://doi.org/10.1007/s40593-022-00301-6

Vancouver

Myers MC, Wilson J. Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm. International Journal of Artificial Intelligence in Education. 2023;33(3):609-634. doi:10.1007/s40593-022-00301-6

BibTeX
@article{Myers2023,
  author = {Myers, Matthew C. and Wilson, Joshua},
  title = {Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm},
  journal = {International Journal of Artificial Intelligence in Education},
  year = {2023},
  volume = {33},
  number = {3},
  pages = {609--634},
  doi = {10.1007/s40593-022-00301-6}
}
Kostenlos starten →