A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions
Sensors · 2023
DOI: 10.3390/s23094373 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, 2023. Verfasst von Muhammad Asif Razzaq und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Fatima Jinnah Women University, Rawalpindi 46000, Pakistan. Hauptthemen laut MEDLINE: Artificial Intelligence, Algorithms. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & communications Technology Planning & Evaluation (IITP), Lean UX core technology and platform for any digital artifacts UX evaluation, Grand Information Technology Research Center support program.
Abstract (Originalsprache)
Multimodal emotion recognition has gained much traction in the field of affective computing, human–computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Fatima Jinnah Women University, Rawalpindi 46000, Pakistan.
- Ubiquitous Computing Lab, Department of Computer Science and Engineering, Kyung Hee University, Seocheon-dong, Giheung-gu, Yongin-si 17104, Republic of Korea.
- Department of Data Science, Sejong University, Seoul 30019, Republic of Korea.
- Hanwha Corporation/Momentum, Hanwha Building, 86 Cheonggyecheon-ro, Jung-gu, Seoul 04541, Republic of Korea.
- Department of Computing, School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan.
ORCID-Profile: Muhammad Asif Razzaq, Jamil Hussain, Jaehun Bang, Cam-Hao Hua, Fahad Ahmed Satti, Ubaid Ur Rehman, Hafiz Syed Muhammad Bilal, Seong Tae Kim, Sungyoung Lee
Förderung
- Institute of Information & communications Technology Planning & Evaluation (IITP)
- Lean UX core technology and platform for any digital artifacts UX evaluation
- Grand Information Technology Research Center support program
Zitieren
APA
Razzaq, M. A., Hussain, J., Bang, J., Hua, C. H., Satti, F. A., Rehman, U. U., Bilal, H. S. M., Kim, S. T., & Lee, S. (2023). A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions. Sensors. https://doi.org/10.3390/s23094373
Vancouver
Razzaq MA, Hussain J, Bang J, Hua CH, Satti FA, Rehman UU, et al. A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions. Sensors. 2023. doi:10.3390/s23094373
BibTeX
@article{Razzaq2023,
author = {Razzaq, Muhammad Asif and Hussain, Jamil and Bang, Jaehun and Hua, Cam-Hao and Satti, Fahad Ahmed and Rehman, Ubaid Ur and Bilal, Hafiz Syed Muhammad and Kim, Seong Tae and Lee, Sungyoung},
title = {A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions},
journal = {Sensors},
year = {2023},
doi = {10.3390/s23094373}
}
EUVIMED