EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Cross Modal Pace Codes for Meeting Speech Separation Under Lip Motion Drift and Audio Visual Synchronization Error

Nuno Molina, Jihyun Park, Miles Douglas, Dohee Hwang

International Journal of Computational and Biological Sciences · 25.06.2026 · 3 (2)

DOI: 10.66238/ijcbs88 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computational and Biological Sciences, online erschienen am 25.06.2026. Verfasst von Nuno Molina und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The integration of visual cues in speech separation systems has significantly advanced the capability to isolate target speakers in complex acoustic environments, commonly referred to as the cocktail party problem. However, the efficacy of audio-visual speech separation models is heavily reliant on the strict temporal synchronization between the audio waveform and the corresponding visual lip movements. In real-world meeting scenarios, teleconferencing platforms frequently experience variable network latency, dynamic video frame drops, and erratic processing bottlenecks, leading to a phenomenon defined as lip motion drift. This drift manifests as non-linear, unpredictable temporal misalignment between modalities, which causes traditional early-fusion and late-fusion models to fail catastrophically. To address this limitation, this paper introduces a novel synchronization and separation paradigm centered on Cross-Modal Pace Codes. By abstracting the rhythmic temporal velocity of both the acoustic envelope and the visual lip articulation into discrete, low-dimensional pace tokens, the proposed system establishes dynamic anchor points across modalities. These pace codes drive a cross-attention alignment network that continuously recalibrates the visual feature stream to match the acoustic target, effectively compensating for both linear latency and non-linear jitter. Extensive evaluations on simulated and real-world meeting datasets demonstrate that the proposed architecture maintains high-fidelity speech separation under severe drift conditions, significantly outperforming state-of-the-art synchronous models. The results highlight the critical importance of rhythm-aware processing in multi-modal systems deployed in unconstrained environments.

Zitieren

APA

Molina, N., Park, J., Douglas, M., & Hwang, D. (2026). Cross Modal Pace Codes for Meeting Speech Separation Under Lip Motion Drift and Audio Visual Synchronization Error. International Journal of Computational and Biological Sciences, 3(2). https://doi.org/10.66238/ijcbs88

Vancouver

Molina N, Park J, Douglas M, Hwang D. Cross Modal Pace Codes for Meeting Speech Separation Under Lip Motion Drift and Audio Visual Synchronization Error. International Journal of Computational and Biological Sciences. 2026;3(2). doi:10.66238/ijcbs88

BibTeX
@article{Molina2026,
  author = {Molina, Nuno and Park, Jihyun and Douglas, Miles and Hwang, Dohee},
  title = {Cross Modal Pace Codes for Meeting Speech Separation Under Lip Motion Drift and Audio Visual Synchronization Error},
  journal = {International Journal of Computational and Biological Sciences},
  year = {2026},
  volume = {3},
  number = {2},
  doi = {10.66238/ijcbs88}
}
Kostenlos starten →