Multimodal Engagement Analysis From Facial Videos in the Classroom
IEEE Transactions on Affective Computing · 01.04.2023 · 14 (2) · 1012-1027
DOI: 10.1109/taffc.2021.3127692 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, erschienen am 01.04.2023. Verfasst von Ömer Sümer und 5 weiteren Autor:innen. Die Arbeit wurde bisher 137-mal zitiert (Crossref) und nennt 96 Literaturangaben. Gefördert von: Ministry of Science, Research and the Arts of the state of Baden Württemberg, Leibniz-WissenschaftsCampus Tübingen, NSF National AI Institute for Student-AI Teaming.
Abstract (Originalsprache)
Student engagement is a key component of learning and teaching, resulting in a plethora of automated methods to measure it. Whereas most of the literature explores student engagement analysis using computer-based learning often in the lab, we focus on using classroom instruction in authentic learning environments. We collected audiovisual recordings of secondary school classes over a one and a half month period, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement from facial videos. We learned deep embeddings for attentional and affective features by training Attention-Net for head pose estimation and Affect-Net for facial expression recognition using previously-collected large-scale datasets. We used these representations to train engagement classifiers on our data, in individual and multiple channel settings, considering temporal dependencies. The best performing engagement classifiers achieved student-independent AUCs of .620 and .720 for grades 8 and 12, respectively, with attention-based features outperforming affective features. Score-level fusion either improved the engagement classifiers or was on par with the best performing modality. We also investigated the effect of personalization and found that only 60 seconds of person-specific data, selected by margin uncertainty of the base classifier, yielded an average AUC improvement of .084.
Förderung
- Ministry of Science, Research and the Arts of the state of Baden Württemberg
- Leibniz-WissenschaftsCampus Tübingen
- NSF National AI Institute for Student-AI Teaming
- National Science Foundation
Zitieren
APA
Sümer, Ö., Goldberg, P., D’Mello, S., Gerjets, P., Trautwein, U., & Kasneci, E. (2023). Multimodal Engagement Analysis From Facial Videos in the Classroom. IEEE Transactions on Affective Computing, 14(2), 1012-1027. https://doi.org/10.1109/taffc.2021.3127692
Vancouver
Sümer Ö, Goldberg P, D’Mello S, Gerjets P, Trautwein U, Kasneci E. Multimodal Engagement Analysis From Facial Videos in the Classroom. IEEE Transactions on Affective Computing. 2023;14(2):1012-1027. doi:10.1109/taffc.2021.3127692
BibTeX
@article{Smer2023,
author = {Sümer, Ömer and Goldberg, Patricia and D’Mello, Sidney and Gerjets, Peter and Trautwein, Ulrich and Kasneci, Enkelejda},
title = {Multimodal Engagement Analysis From Facial Videos in the Classroom},
journal = {IEEE Transactions on Affective Computing},
year = {2023},
volume = {14},
number = {2},
pages = {1012--1027},
doi = {10.1109/taffc.2021.3127692}
}
EUVIMED