EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS
International Journal on Integrating Technology in Education · 28.03.2026 · 15 (1) · 37-49
DOI: 10.5121/ijite.2026.15103 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal on Integrating Technology in Education, online erschienen am 28.03.2026. Verfasst von Ryuya Itano und 5 weiteren Autor:innen.
Abstract (Originalsprache)
Crowdsourcing assumes a transient relationship between task requesters and workers, which makes it hard for workers to improve their skills. In addition, crowdsourcingtasksare shifting from simple to more complex and open-ended, highlighting the importance of training workers to handle such tasks. Although various methods have been proposed to train workers, a method to evaluate their skill levelsin open-ended tasks has not yet been established. Direct evaluation by requesters is desirable, but scaling up tasks is difficult due to the requester’s heavy workload.This study aims to explore methods for evaluating workers without increasing requesters’ workload, comparing and verifying apeer-basedmethod and anLLM-based automated method.The experiment investigated the alignment between evaluations fromthe twomethods andthose from requesters, thereby clarifying the characteristics of each method.The experimental resultsdemonstrated the applicability of LLMs to evaluating workers in open-ended tasks, revealing both their strengths in consistency and limitations in capturing subtle human judgments
Zitieren
APA
Itano, R., Tanitsu, H., Bamba, M., Noseyama, R., Kohiga, A., & Koita, T. (2026). EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS. International Journal on Integrating Technology in Education, 15(1), 37-49. https://doi.org/10.5121/ijite.2026.15103
Vancouver
Itano R, Tanitsu H, Bamba M, Noseyama R, Kohiga A, Koita T. EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS. International Journal on Integrating Technology in Education. 2026;15(1):37-49. doi:10.5121/ijite.2026.15103
BibTeX
@article{Itano2026,
author = {Itano, Ryuya and Tanitsu, Honoka and Bamba, Motoki and Noseyama, Ryota and Kohiga, Akihito and Koita, Takahiro},
title = {EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS},
journal = {International Journal on Integrating Technology in Education},
year = {2026},
volume = {15},
number = {1},
pages = {37--49},
doi = {10.5121/ijite.2026.15103}
}
EUVIMED