Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications
Nordic Machine Intelligence · 2025
DOI: 10.5617/nmi.12537 ↗Kurz gefasst
Fachartikel in der Zeitschrift Nordic Machine Intelligence, 2025. Verfasst von Pegah Salehi und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Dynamic facial emotion is essential for believable AI-generated avatars; however, most systems remain visually inert, limiting their utility in simulations such as virtual training for investigative interviews with abused children. We introduce and evaluate a real-time architecture fusing Unreal Engine 5 MetaHuman rendering with NVIDIA Omniverse Audio2Face to translate vocal prosody into high-fidelity facial expressions on photorealistic child avatars. Due to limitations in synthetic voice options, both avatars were voiced using a young adult female TTS model, selected from two different systems in an attempt to better match each character. This compromise introduces a confounding factor. Voice-age mismatches and prosodic or emotional differences may disrupt audiovisual alignment. We implemented a two-PC setup decoupling language and speech synthesis from GPU-intensive rendering, designed to support low-latency interaction in desktop and VR environments. A between-subjects study ($N=70$) using audio+visual and visual-only conditions assessed perceptual impacts as participants rated emotional clarity, facial realism, and empathy for two avatars expressing joy, sadness, and anger. Our results show that while avatars could express emotions recognizably, particularly sadness and joy, recognition of anger dropped markedly without audio, highlighting the role of vocal cues in conveying high-arousal states. Interestingly, silencing the clips improved perceived realism by removing mismatches between facial animation and voice, especially where age or emotional tone were incongruent. These findings highlight that perceived believability hinges on the interplay between audiovisual congruence and facial geometry: a mismatched voice can undermine even well-crafted expressions, while a congruent one can enhance weaker visuals. This trade-off presents an ongoing challenge for designing emotionally coherent avatars in sensitive training contexts.
Zitieren
APA
Salehi, P., Sheshkal, S. A., Thambawita, V., Riegler, M. A., & Halvorsen, P. (2025). Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications. Nordic Machine Intelligence. https://doi.org/10.5617/nmi.12537
Vancouver
Salehi P, Sheshkal SA, Thambawita V, Riegler MA, Halvorsen P. Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications. Nordic Machine Intelligence. 2025. doi:10.5617/nmi.12537
BibTeX
@article{Salehi2025,
author = {Salehi, Pegah and Sheshkal, Sajad Amouei and Thambawita, Vajira and Riegler, Michael A. and Halvorsen, Pål},
title = {Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications},
journal = {Nordic Machine Intelligence},
year = {2025},
doi = {10.5617/nmi.12537}
}
EUVIMED