EUVIMEDEUVIMED entdecken
Crossref

EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS

Ryuya Itano, Honoka Tanitsu, Motoki Bamba, Ryota Noseyama, Akihito Kohiga, Takahiro Koita

International Journal on Integrating Technology in Education · 28.03.2026 · 15 (1) · 37-49

DOI: 10.5121/ijite.2026.15103 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal on Integrating Technology in Education, online erschienen am 28.03.2026. Verfasst von Ryuya Itano und 5 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Crowdsourcing assumes a transient relationship between task requesters and workers, which makes it hard for workers to improve their skills. In addition, crowdsourcingtasksare shifting from simple to more complex and open-ended, highlighting the importance of training workers to handle such tasks. Although various methods have been proposed to train workers, a method to evaluate their skill levelsin open-ended tasks has not yet been established. Direct evaluation by requesters is desirable, but scaling up tasks is difficult due to the requester’s heavy workload.This study aims to explore methods for evaluating workers without increasing requesters’ workload, comparing and verifying apeer-basedmethod and anLLM-based automated method.The experiment investigated the alignment between evaluations fromthe twomethods andthose from requesters, thereby clarifying the characteristics of each method.The experimental resultsdemonstrated the applicability of LLMs to evaluating workers in open-ended tasks, revealing both their strengths in consistency and limitations in capturing subtle human judgments

Zitieren

APA

Itano, R., Tanitsu, H., Bamba, M., Noseyama, R., Kohiga, A., & Koita, T. (2026). EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS. International Journal on Integrating Technology in Education, 15(1), 37-49. https://doi.org/10.5121/ijite.2026.15103

Vancouver

Itano R, Tanitsu H, Bamba M, Noseyama R, Kohiga A, Koita T. EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS. International Journal on Integrating Technology in Education. 2026;15(1):37-49. doi:10.5121/ijite.2026.15103

BibTeX
@article{Itano2026,
  author = {Itano, Ryuya and Tanitsu, Honoka and Bamba, Motoki and Noseyama, Ryota and Kohiga, Akihito and Koita, Takahiro},
  title = {EXPLORING CROWDSOURCED WORKER EVALUATION METHODSIN OPEN-ENDED TASKS},
  journal = {International Journal on Integrating Technology in Education},
  year = {2026},
  volume = {15},
  number = {1},
  pages = {37--49},
  doi = {10.5121/ijite.2026.15103}
}
Kostenlos starten →