Leveraging Multimodal Large Language Models to Analyse Student Exploration Behaviours in Educational Game Environments
Journal of Computer Assisted Learning · 2026-08 · 42 (4)
DOI: 10.1002/jcal.70269 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computer Assisted Learning, online erschienen am 03.06.2026. Verfasst von Yiqiu Zhou und 7 weiteren Autor:innen. Die Arbeit nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Background
Video data provides rich opportunities to examine student behaviour in game‐based learning environments, capturing not only observable actions but also subtle indicators of cognitive engagement. However, traditional video analysis is labor‐intensive, and current applications of AI to this task are limited by models trained on general‐purpose datasets, which often fail to capture the pedagogical meaning of student actions in authentic educational contexts.
Objectives
This study explores how multimodal large language models (MLLMs), specifically LLaVA‐Video‐7B‐Qwen2, can support qualitative video analysis of student exploration behaviours in a Minecraft‐based STEM learning environment.
Methods
We conducted an exploratory case study using screen recordings from a Minecraft‐based STEM learning environment. We tested multiple prompt strategies for guiding MLLM‐generated video descriptions and found that role‐assignment prompting performed most effectively. We evaluated model outputs using a mixed‐method framework that included quantitative scoring, GPT‐based judgement, and researcher validation. Results and Conclusions Our findings show that MLLMs can reliably identify surface‐level behaviours, such as navigation patterns and object interactions, but struggle to infer the intent or goals behind student actions, leading to a significant rate of over‐interpretation (26.5% when explaining student strategies). The model's outputs are sensitive to prompt phrasing, underscoring the importance of prompt engineering. While current MLLMs show promise for streamlining parts of the video analysis workflow, their use in educational contexts requires structured oversight and careful interpretation to ensure reliability and relevance.
Förderung
- National Science Foundation
Zitieren
APA
Zhou, Y., Liu, X., Zambrano, A. F., Paquette, L., Barany, A., Ocumpaugh, J., Baker, R. S., & Ginger, J. (2026). Leveraging Multimodal Large Language Models to Analyse Student Exploration Behaviours in Educational Game Environments. Journal of Computer Assisted Learning, 42(4). https://doi.org/10.1002/jcal.70269
Vancouver
Zhou Y, Liu X, Zambrano AF, Paquette L, Barany A, Ocumpaugh J, et al. Leveraging Multimodal Large Language Models to Analyse Student Exploration Behaviours in Educational Game Environments. Journal of Computer Assisted Learning. 2026;42(4). doi:10.1002/jcal.70269
BibTeX
@article{Zhou2026,
author = {Zhou, Yiqiu and Liu, Xiner and Zambrano, Andres Felipe and Paquette, Luc and Barany, Amanda and Ocumpaugh, Jaclyn and Baker, Ryan S. and Ginger, Jeff},
title = {Leveraging Multimodal Large Language Models to Analyse Student Exploration Behaviours in Educational Game Environments},
journal = {Journal of Computer Assisted Learning},
year = {2026},
volume = {42},
number = {4},
doi = {10.1002/jcal.70269}
}
EUVIMED