Measuring the Reliability of a Gamified Stroop Task: Quantitative Experiment
JMIR Serious Games · 10.04.2024 · 12 · e50315
DOI: 10.2196/50315 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Serious Games, online erschienen am 10.04.2024. Verfasst von Katelyn Wiley und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, University of Saskatchewan, Saskatoon, SK, Canada. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Few gamified cognitive tasks are subjected to rigorous examination of psychometric properties, despite their use in experimental and clinical settings. Even small manipulations to cognitive tasks require extensive research to understand their effects.
Objective
This study aims to investigate how game elements can affect the reliability of scores on a Stroop task. We specifically investigated performance consistency within and across sessions.
Methods
We created 2 versions of the Stroop task, with and without game elements, and then tested each task with participants at 2 time points. The gamified task used points and feedback as game elements. In this paper, we report on the reliability of the gamified Stroop task in terms of internal consistency and test-retest reliability, compared with the control task. We used a permutation approach to evaluate internal consistency. For test-retest reliability, we calculated the Pearson correlation and intraclass correlation coefficients between each time point. We also descriptively compared the reliability of scores on a trial-by-trial basis, considering the different trial types.
Results
At the first time point, the Stroop effect was reduced in the game condition, indicating an increase in performance. Participants in the game condition had faster reaction times (P=.005) and lower error rates (P=.04) than those in the basic task condition. Furthermore, the game condition led to higher measures of internal consistency at both time points for reaction times and error rates, which indicates a more consistent response pattern. For reaction time in the basic task condition, at time 1, rSpearman-Brown=0.78, 95% CI 0.64-0.89. At time 2, rSpearman-Brown=0.64, 95% CI 0.40-0.81. For reaction time, in the game condition, at time 1, rSpearman-Brown=0.83, 95% CI 0.71-0.91. At time 2, rSpearman-Brown=0.76, 95% CI 0.60-0.88. Similarly, for error rates in the basic task condition, at time 1, rSpearman-Brown=0.76, 95% CI 0.62-0.87. At time 2, rSpearman-Brown=0.74, 95% CI 0.58-0.86. For error rates in the game condition, at time 1, rSpearman-Brown=0.76, 95% CI 0.62-0.87. At time 2, rSpearman-Brown=0.74, 95% CI 0.58-0.86. Test-retest reliability analysis revealed a distinctive performance pattern depending on the trial type, which may be reflective of motivational differences between task versions. In short, especially in the incongruent trials where cognitive conflict occurs, performance in the game condition reaches peak consistency after 100 trials, whereas performance consistency drops after 50 trials for the basic version and only catches up to the game after 250 trials.
Conclusions
Even subtle gamification can impact task performance albeit not only in terms of a direct difference in performance between conditions. People playing the game reach peak performance sooner, and their performance is more consistent within and across sessions. We advocate for a closer examination of the impact of game elements on performance.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, University of Saskatchewan, Saskatoon, SK, Canada.
- Faculty of Behavioural, Management and Social Sciences, University of Twente, Enschede, Netherlands.
- School of Psychology, University College Dublin, Dublin, Ireland.
- Max-Planck-Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
- Department of Computer Science, University of Victoria, Victoria, BC, Canada.
ORCID-Profile: Katelyn Wiley, Phaedra Berger, Maximilian Achim Friehs, Regan Lee Mandryk
Zitieren
APA
Wiley, K., Berger, P., Friehs, M. A., & Mandryk, R. L. (2024). Measuring the Reliability of a Gamified Stroop Task: Quantitative Experiment. JMIR Serious Games, 12, e50315. https://doi.org/10.2196/50315
Vancouver
Wiley K, Berger P, Friehs MA, Mandryk RL. Measuring the Reliability of a Gamified Stroop Task: Quantitative Experiment. JMIR Serious Games. 2024;12:e50315. doi:10.2196/50315
BibTeX
@article{Wiley2024,
author = {Wiley, Katelyn and Berger, Phaedra and Friehs, Maximilian Achim and Mandryk, Regan Lee},
title = {Measuring the Reliability of a Gamified Stroop Task: Quantitative Experiment},
journal = {JMIR Serious Games},
year = {2024},
volume = {12},
pages = {e50315},
doi = {10.2196/50315}
}
EUVIMED