Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models
IEEE Access · 2024 · 12 · 20692-20704
DOI: 10.1109/access.2024.3362247 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2024. Verfasst von Muhammad Tahir Naseem und 2 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Basic Science Research Program through the National Research Foundation of Korea, Ministry of Education, Ministry of Trade, Industry and Energy (MOTIE), South Korea, through the Project “Development of convergence lighting technology based on affective perception….
Abstract (Originalsprache)
Facial expression recognition (FER) is one of the best non-intrusive methods for understanding and tracking mood and mental states. In this study, we propose early and late fusion methods to recognize five facial expressions (angry, happy, neutral, sad, and surprised) using different combinations from a publicly available database (VIRI) with visible, infrared, and multispectral dynamic imaging (MSX) images and the (NVIE) database. A distinctive feature is the use of concatenation and combining techniques to combine ResNet-18 with transfer learning (TL) to create a model that is significantly more accurate than individual models. In the early fusion, we concatenated features from the modalities and classified facial expressions (FEs). In the late fusion, we combined the outputs of the modalities using weighted sums. For this purpose, we used different weighting factors depending on the accuracy of the individual models. The experimental results demonstrated that the proposed model outperformed the previous works by providing an accuracy of 83.33% when we trained the model (1-step training). Through further fine-tuning (3-step training), we obtained an improved performance of 84.44%. We conducted additional experiments by combining them with another modality (MSX) available in the database. By performing experiments with an additional modality (MSX), we obtained improved performance, which confirms that the additional modality combined with existing modalities can help improve the performance of fusion models for facial expression recognition. We also experimented by changing the backbones (Vgg-16, ShuffleNetv2, MobileNetv2, and GhostNet) in addition to ResNet-18 for visible and MSX data. ResNet-18 outperformed the other backbones in facial expression recognition for visible and MSX data.
Förderung
- Basic Science Research Program through the National Research Foundation of Korea
- Ministry of Education
- Ministry of Trade, Industry and Energy (MOTIE), South Korea, through the Project “Development of convergence lighting technology based on affective perception…
Zitieren
APA
Naseem, M. T., Lee, C., & Kim, N. (2024). Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models. IEEE Access, 12, 20692-20704. https://doi.org/10.1109/access.2024.3362247
Vancouver
Naseem MT, Lee C, Kim N. Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models. IEEE Access. 2024;12:20692-20704. doi:10.1109/access.2024.3362247
BibTeX
@article{Naseem2024,
author = {Naseem, Muhammad Tahir and Lee, Chan-Su and Kim, Na-Hyun},
title = {Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models},
journal = {IEEE Access},
year = {2024},
volume = {12},
pages = {20692--20704},
doi = {10.1109/access.2024.3362247}
}
EUVIMED