Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion
IEEE Transactions on Affective Computing · 01.04.2023 · 14 (2) · 1236-1248
DOI: 10.1109/taffc.2021.3122146 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, erschienen am 01.04.2023. Verfasst von Fuyan Ma und 2 weiteren Autor:innen. Die Arbeit wurde bisher 308-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Key Research and Development Project, National Natural Science Foundation of China, Key-Area Research and Development Plan of Guangdong Province.
Abstract (Originalsprache)
Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.
Förderung
- National Key Research and Development Project
- National Natural Science Foundation of China
- Key-Area Research and Development Plan of Guangdong Province
Zitieren
APA
Ma, F., Sun, B., & Li, S. (2023). Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion. IEEE Transactions on Affective Computing, 14(2), 1236-1248. https://doi.org/10.1109/taffc.2021.3122146
Vancouver
Ma F, Sun B, Li S. Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion. IEEE Transactions on Affective Computing. 2023;14(2):1236-1248. doi:10.1109/taffc.2021.3122146
BibTeX
@article{Ma2023,
author = {Ma, Fuyan and Sun, Bin and Li, Shutao},
title = {Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion},
journal = {IEEE Transactions on Affective Computing},
year = {2023},
volume = {14},
number = {2},
pages = {1236--1248},
doi = {10.1109/taffc.2021.3122146}
}
EUVIMED