EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Personality Recognition Using Self-Attention-Based Fusion of Audio, Visual, and Text Features

Hyeonuk Bhin, Jongsuk Choi

Electronics · 15.07.2025 · 14 (14) · 2837

DOI: 10.3390/electronics14142837 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 15.07.2025. Verfasst von Hyeonuk Bhin und 1 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Korea Institute of Science and Technology (KIST) Institutional Program, Ministry of Trade Industry & Energy (MOTIE, Korea).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Personality is a fundamental psychological trait that exerts a long-term influence on human behavior patterns and social interactions. Automatic personality recognition (APR) has exhibited increasing importance across various domains, including Human–Robot Interaction (HRI), personalized services, and psychological assessments. In this study, we propose a multimodal personality recognition model that classifies the Big Five personality traits by extracting features from three heterogeneous sources: audio processed using Wav2Vec2, video represented as Skeleton Landmark time series, and text encoded through Bidirectional Encoder Representations from Transformers (BERT) and Doc2Vec embeddings. Each modality is handled through an independent Self-Attention block that highlights salient temporal information, and these representations are then summarized and integrated using a late fusion approach to effectively reflect both the inter-modal complementarity and cross-modal interactions. Compared to traditional recurrent neural network (RNN)-based multimodal models and unimodal classifiers, the proposed model achieves an improvement of up to 12 percent in the F1-score. It also maintains a high prediction accuracy and robustness under limited input conditions. Furthermore, a visualization based on t-distributed Stochastic Neighbor Embedding (t-SNE) demonstrates clear distributional separation across the personality classes, enhancing the interpretability of the model and providing insights into the structural characteristics of its latent representations. To support real-time deployment, a lightweight thread-based processing architecture is implemented, ensuring computational efficiency. By leveraging deep learning-based feature extraction and the Self-Attention mechanism, we present a novel personality recognition framework that balances performance with interpretability. The proposed approach establishes a strong foundation for practical applications in HRI, counseling, education, and other interactive systems that require personalized adaptation.

Förderung

  • Korea Institute of Science and Technology (KIST) Institutional Program
  • Ministry of Trade Industry & Energy (MOTIE, Korea)

Zitieren

APA

Bhin, H., & Choi, J. (2025). Multimodal Personality Recognition Using Self-Attention-Based Fusion of Audio, Visual, and Text Features. Electronics, 14(14), 2837. https://doi.org/10.3390/electronics14142837

Vancouver

Bhin H, Choi J. Multimodal Personality Recognition Using Self-Attention-Based Fusion of Audio, Visual, and Text Features. Electronics. 2025;14(14):2837. doi:10.3390/electronics14142837

BibTeX
@article{Bhin2025,
  author = {Bhin, Hyeonuk and Choi, Jongsuk},
  title = {Multimodal Personality Recognition Using Self-Attention-Based Fusion of Audio, Visual, and Text Features},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {14},
  pages = {2837},
  doi = {10.3390/electronics14142837}
}
Kostenlos starten →