U-Shaped Distribution Guided Sign Language Emotion Recognition With Semantic and Movement Features
IEEE Transactions on Affective Computing · 2024-10 · 15 (4) · 2180-2191
DOI: 10.1109/taffc.2024.3409357 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2024-10. Verfasst von Jiangtao Zhang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 63 Literaturangaben. Gefördert von: Anhui Provincial Natural Science Foundation of China.
Abstract (Originalsprache)
Emotional expression is a bridge to human communication, especially for the hearing impaired. This paper proposes a sign language emotion recognition method based on semantic and movement features by exploring the relationship between emotion valence and arousal in-depth, called SeMER. The SeMER framework includes a semantic extractor, a movement feature extractor, and an emotion classifier. The contextual relations obtained from the sign language recognition task are added to the semantic extractor as prior knowledge using a transfer learning approach to better acquire the affective polarity of semantics. In the movement feature extractor based on graph convolutional networks, a spatial-temporal adjacency matrix of gestures and node attention matrix are developed to aggregate the emotion-related movement features of intra- and inter-gestures. The proposed emotion classifier maps semantic and movement features to the emotion space. The validated U-shaped distributions of valance and arousal are then used to guide the relationship between them, and improve the accuracy of emotion prediction. In addition, a sign language emotion dataset containing 5 emotions from 18 participants, SE-Sentence, is collected through armbands with built-in surface electromyograph and inertial measurement unit sensors. Experimental results showed that SeMER achieved an accuracy and f1 value of 88% on SE-Sentence.
Förderung
- Anhui Provincial Natural Science Foundation of China
Zitieren
APA
Zhang, J., Wang, Q., & Wang, Q. (2024). U-Shaped Distribution Guided Sign Language Emotion Recognition With Semantic and Movement Features. IEEE Transactions on Affective Computing, 15(4), 2180-2191. https://doi.org/10.1109/taffc.2024.3409357
Vancouver
Zhang J, Wang Q, Wang Q. U-Shaped Distribution Guided Sign Language Emotion Recognition With Semantic and Movement Features. IEEE Transactions on Affective Computing. 2024;15(4):2180-2191. doi:10.1109/taffc.2024.3409357
BibTeX
@article{Zhang2024,
author = {Zhang, Jiangtao and Wang, Qingshan and Wang, Qi},
title = {U-Shaped Distribution Guided Sign Language Emotion Recognition With Semantic and Movement Features},
journal = {IEEE Transactions on Affective Computing},
year = {2024},
volume = {15},
number = {4},
pages = {2180--2191},
doi = {10.1109/taffc.2024.3409357}
}
EUVIMED