Unified Visual Synchrony: A Framework for Face–Gesture Coherence in Multimodal Human–AI Interaction
Big Data and Cognitive Computing · 12.03.2026 · 10 (3) · 88
DOI: 10.3390/bdcc10030088 ↗Kurz gefasst
Fachartikel in der Zeitschrift Big Data and Cognitive Computing, online erschienen am 12.03.2026. Verfasst von Saule Kudubayeva und 6 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan.
Abstract (Originalsprache)
Multimodal human–AI systems generally consider facial expressions and body motions as separate input streams, leading to disjointed interpretations and diminished emotional coherence. To overcome this issue, we offer the Engagement-Safe Expressive Alignment (ESEA) paradigm and the Unified Visual Synchrony (UVS) framework as its computational implementation. UVS models the coherence between facial expressions and gestures, offering an interpretable visual synchrony signal that can function as adaptive feedback in human–AI interactions. The framework’s key component is the Consistency Index for Affective Synchrony (CIAS), which correlates brief visual segments with scalar synchrony scores through a common latent representation. Facial and gestural signals are processed by modality-specific projection networks into a unified latent space, and CIAS is derived from the similarity and short-term temporal consistency of these latent trajectories. The synchrony index is regarded as an estimation of affective visual coherence within the ESEA paradigm. We formalize the UVS/CIAS framework and conduct a comparative experimental evaluation utilizing matched and mismatched face–gesture segments derived from rendered dialog footage. Utilizing ROC analysis, score distribution comparisons, temporal visualizations, and negative control tests, we illustrate that CIAS effectively captures structured face–gesture alignment that surpasses similarity-based baselines, while also delivering a persistent, time-resolved synchronization signal. These findings establish CIAS as a principled and interpretable feedback signal for future affect-aware, engagement-focused multimodal agents.
Förderung
- Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan
Zitieren
APA
Kudubayeva, S., Seksenbayev, Y., Yerimbetova, A., Daiyrbayeva, E., Sakenov, B., Telman, D., & Turdalyuly, M. (2026). Unified Visual Synchrony: A Framework for Face–Gesture Coherence in Multimodal Human–AI Interaction. Big Data and Cognitive Computing, 10(3), 88. https://doi.org/10.3390/bdcc10030088
Vancouver
Kudubayeva S, Seksenbayev Y, Yerimbetova A, Daiyrbayeva E, Sakenov B, Telman D, et al. Unified Visual Synchrony: A Framework for Face–Gesture Coherence in Multimodal Human–AI Interaction. Big Data and Cognitive Computing. 2026;10(3):88. doi:10.3390/bdcc10030088
BibTeX
@article{Kudubayeva2026,
author = {Kudubayeva, Saule and Seksenbayev, Yernar and Yerimbetova, Aigerim and Daiyrbayeva, Elmira and Sakenov, Bakzhan and Telman, Duman and Turdalyuly, Mussa},
title = {Unified Visual Synchrony: A Framework for Face–Gesture Coherence in Multimodal Human–AI Interaction},
journal = {Big Data and Cognitive Computing},
year = {2026},
volume = {10},
number = {3},
pages = {88},
doi = {10.3390/bdcc10030088}
}
EUVIMED