EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models

Ali Darvishi, Hassan Khosravi, Afshin Rahimi, Shazia Sadiq, Dragan Gasevic

IEEE Transactions on Learning Technologies · 01.02.2023 · 16 (1) · 106-120

DOI: 10.1109/tlt.2022.3229022 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Learning Technologies, erschienen am 01.02.2023. Verfasst von Ali Darvishi und 4 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 102 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Australian Research Council's Industrial Transformation Training Centre for Information Resilience.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilize a repository of student-generated content (SGC), a selection process is needed to separate high- from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach is to use a peer-review process where students are asked to assess the quality of resources authored by their peers. Given that judgments of students, as experts-in-training, cannot wholly be relied upon, a redundancy-based method is widely employed where the same assessment task is given to multiple students. However, this approach introduces a new challenge, referred to as the consensus problem: How can we assign a final quality to a resource given ratings by multiple students? To address this challenge, we investigate the predictive performance of 18 inference models across five well-established categories of consensus approaches for inferring the quality of SGC at scale. The analysis is based on the engagement of 2141 undergraduate students across five courses in creating 12 803 resources and 77 297 peer reviews. Results indicate that the quality of reviews is quite diverse, and students tend to overrate. Consequently, simple statistics such as mean and median fail to identify poor-quality resources. Findings further suggest that incorporating advanced probabilistic and text analysis methods to infer the reviewers' reliability and reviews' quality improves performance; however, there is still an evident need for instructor oversight and training of students to write compelling and reliable reviews.

Förderung

  • Australian Research Council's Industrial Transformation Training Centre for Information Resilience

Zitieren

APA

Darvishi, A., Khosravi, H., Rahimi, A., Sadiq, S., & Gasevic, D. (2023). Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models. IEEE Transactions on Learning Technologies, 16(1), 106-120. https://doi.org/10.1109/tlt.2022.3229022

Vancouver

Darvishi A, Khosravi H, Rahimi A, Sadiq S, Gasevic D. Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models. IEEE Transactions on Learning Technologies. 2023;16(1):106-120. doi:10.1109/tlt.2022.3229022

BibTeX
@article{Darvishi2023,
  author = {Darvishi, Ali and Khosravi, Hassan and Rahimi, Afshin and Sadiq, Shazia and Gasevic, Dragan},
  title = {Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models},
  journal = {IEEE Transactions on Learning Technologies},
  year = {2023},
  volume = {16},
  number = {1},
  pages = {106--120},
  doi = {10.1109/tlt.2022.3229022}
}
Kostenlos starten →