EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications

Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Michael A. Riegler, Pål Halvorsen

Nordic Machine Intelligence · 2025

DOI: 10.5617/nmi.12537 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Nordic Machine Intelligence, 2025. Verfasst von Pegah Salehi und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Dynamic facial emotion is essential for believable AI-generated avatars; however, most systems remain visually inert, limiting their utility in simulations such as virtual training for investigative interviews with abused children. We introduce and evaluate a real-time architecture fusing Unreal Engine 5 MetaHuman rendering with NVIDIA Omniverse Audio2Face to translate vocal prosody into high-fidelity facial expressions on photorealistic child avatars. Due to limitations in synthetic voice options, both avatars were voiced using a young adult female TTS model, selected from two different systems in an attempt to better match each character. This compromise introduces a confounding factor. Voice-age mismatches and prosodic or emotional differences may disrupt audiovisual alignment. We implemented a two-PC setup decoupling language and speech synthesis from GPU-intensive rendering, designed to support low-latency interaction in desktop and VR environments. A between-subjects study ($N=70$) using audio+visual and visual-only conditions assessed perceptual impacts as participants rated emotional clarity, facial realism, and empathy for two avatars expressing joy, sadness, and anger. Our results show that while avatars could express emotions recognizably, particularly sadness and joy, recognition of anger dropped markedly without audio, highlighting the role of vocal cues in conveying high-arousal states. Interestingly, silencing the clips improved perceived realism by removing mismatches between facial animation and voice, especially where age or emotional tone were incongruent. These findings highlight that perceived believability hinges on the interplay between audiovisual congruence and facial geometry: a mismatched voice can undermine even well-crafted expressions, while a congruent one can enhance weaker visuals. This trade-off presents an ongoing challenge for designing emotionally coherent avatars in sensitive training contexts.

Zitieren

APA

Salehi, P., Sheshkal, S. A., Thambawita, V., Riegler, M. A., & Halvorsen, P. (2025). Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications. Nordic Machine Intelligence. https://doi.org/10.5617/nmi.12537

Vancouver

Salehi P, Sheshkal SA, Thambawita V, Riegler MA, Halvorsen P. Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications. Nordic Machine Intelligence. 2025. doi:10.5617/nmi.12537

BibTeX
@article{Salehi2025,
  author = {Salehi, Pegah and Sheshkal, Sajad Amouei and Thambawita, Vajira and Riegler, Michael A. and Halvorsen, Pål},
  title = {Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications},
  journal = {Nordic Machine Intelligence},
  year = {2025},
  doi = {10.5617/nmi.12537}
}
Kostenlos starten →