Brain-Machine Coupled Learning Method for Facial Emotion Recognition
IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.09.2023 · 45 (9) · 10703-10717
DOI: 10.1109/tpami.2023.3257846 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.09.2023. Verfasst von Dongjun Liu und 5 weiteren Autor:innen. Hauptthemen laut MEDLINE: Algorithms, Facial Recognition. Die Arbeit wurde bisher 75-mal zitiert (Crossref) und nennt 103 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Key R&D Program of China, Key Research and Development Project of Zhejiang Province.
Abstract (Originalsprache)
Neural network models of machine learning have shown promising prospects for visual tasks, such as facial emotion recognition (FER). However, the generalization of the model trained from a dataset with a few samples is limited. Unlike the machine, the human brain can effectively realize the required information from a few samples to complete the visual tasks. To learn the generalization ability of the brain, in this article, we propose a novel brain-machine coupled learning method for facial emotion recognition to let the neural network learn the visual knowledge of the machine and cognitive knowledge of the brain simultaneously. The proposed method utilizes visual images and electroencephalogram (EEG) signals to couple training the models in the visual and cognitive domains. Each domain model consists of two types of interactive channels, common and private. Since the EEG signals can reflect brain activity, the cognitive process of the brain is decoded by a model following reverse engineering. Decoding the EEG signals induced by the facial emotion images, the common channel in the visual domain can approach the cognitive process in the cognitive domain. Moreover, the knowledge specific to each domain is found in each private channel using an adversarial strategy. After learning, without the participation of the EEG signals, only the concatenation of both channels in the visual domain is used to classify facial emotion images based on the visual knowledge of the machine and the cognitive knowledge learned from the brain. Experiments demonstrate that the proposed method can produce excellent performance on several public datasets. Further experiments show that the proposed method trained from the EEG signals has good generalization ability on new datasets and can be applied to other network models, illustrating the potential for practical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Natural Science Foundation of China
- National Key R&D Program of China
- Key Research and Development Project of Zhejiang Province
- Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province
- National Key Research and Development Program of China
- Zhejiang Provincial Natural Science Foundation of China
Zitieren
APA
Liu, D., Dai, W., Zhang, H., Jin, X., Cao, J., & Kong, W. (2023). Brain-Machine Coupled Learning Method for Facial Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10703-10717. https://doi.org/10.1109/tpami.2023.3257846
Vancouver
Liu D, Dai W, Zhang H, Jin X, Cao J, Kong W. Brain-Machine Coupled Learning Method for Facial Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(9):10703-10717. doi:10.1109/tpami.2023.3257846
BibTeX
@article{Liu2023,
author = {Liu, Dongjun and Dai, Weichen and Zhang, Hangkui and Jin, Xuanyu and Cao, Jianting and Kong, Wanzeng},
title = {Brain-Machine Coupled Learning Method for Facial Emotion Recognition},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {9},
pages = {10703--10717},
doi = {10.1109/tpami.2023.3257846}
}
EUVIMED