A Class‐Prior‐Adjusted Temporal Multi‐Expert Network for Word‐Level Lip Reading
Expert Systems · 2026-12 · 43 (12)
DOI: 10.1111/exsy.70474 ↗Kurz gefasst
Fachartikel in der Zeitschrift Expert Systems, online erschienen am 08.11.2026. Verfasst von Xinyu Liu und 4 weiteren Autor:innen. Die Arbeit nennt 51 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Lip reading aims to recognize speech from visual lip movements, requiring models to capture subtle variations in lip shape and complex temporal dependencies. Compared with sentence‐level lip reading, word‐level lip reading offers limited linguistic and semantic context and therefore depends more heavily on fine‐grained, short‐term motion cues. Many existing methods use a shared visual backbone followed by a single temporal modelling branch and a unified classifier. This single‐path decision architecture may limit their flexibility in modelling heterogeneous word‐level temporal dynamics. To address this limitation, we propose a class‐prior‐adjusted temporal multi‐expert network (CP‐TME) for word‐level lip reading. CP‐TME employs a two‐stage training strategy that separates shared visual representation learning from multi‐expert temporal decision modelling. In the first stage, we adopt PaCo‐based contrastive pre‐training, which combines learnable parametric class representatives with a dynamic labelled feature queue, to learn discriminative shared visual representations. In the second stage, the pre‐trained VideoCNN backbone is frozen, while multiple experts with independent temporal branches and classification heads are trained using class‐prior logit adjustments derived from empirical class frequencies. Negative and positive prior exponents impose opposing class‐wise logit shifts, whereas the zero exponent preserves the unadjusted logits. These three conditions produce different class‐wise probability distributions and branch‐specific optimization gradients, driving the experts toward distinct temporal decision functions. CP‐TME achieves recognition accuracies of 60.37% on LRW‐1000 and 91.81% on LRW, demonstrating its effectiveness for word‐level lip reading.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Liu, X., Zhang, J., Li, S., Li, M., & Xue, F. (2026). A Class‐Prior‐Adjusted Temporal Multi‐Expert Network for Word‐Level Lip Reading. Expert Systems, 43(12). https://doi.org/10.1111/exsy.70474
Vancouver
Liu X, Zhang J, Li S, Li M, Xue F. A Class‐Prior‐Adjusted Temporal Multi‐Expert Network for Word‐Level Lip Reading. Expert Systems. 2026;43(12). doi:10.1111/exsy.70474
BibTeX
@article{Liu2026,
author = {Liu, Xinyu and Zhang, Jinrui and Li, Shujie and Li, Mengyuan and Xue, Feng},
title = {A Class‐Prior‐Adjusted Temporal Multi‐Expert Network for Word‐Level Lip Reading},
journal = {Expert Systems},
year = {2026},
volume = {43},
number = {12},
doi = {10.1111/exsy.70474}
}
EUVIMED