A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model
Sensors · 2025
DOI: 10.3390/s25082366 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, 2025. Verfasst von Xiangliang Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: The State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China. Hauptthemen laut MEDLINE: Speech Recognition Software, Speech. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSFC, National Key Laboratory of Transient Impact, A Project Supported by Scientific Research Fund of Zhejiang University.
Abstract (Originalsprache)
Visual speech recognition is a technology that relies on visual information, offering unique advantages in noisy environments or when communicating with individuals with speech impairments. However, this technology still faces challenges, such as limited generalization ability due to different speech habits, high recognition error rates caused by confusable phonemes, and difficulties adapting to complex lighting conditions and facial occlusions. This paper proposes a lip reading data augmentation method—Partition-Time Masking (PTM)—to address these challenges and improve lip reading models’ performance and generalization ability. Applying nonlinear transformations to the training data enhances the model’s generalization ability when handling diverse speakers and environmental conditions. A lip-reading recognition model architecture, Swin Transformer and 3D Convolution (ST3D), was designed to overcome the limitations of traditional lip-reading models that use ResNet-based front-end feature extraction networks. By adopting a strategy that combines Swin Transformer and 3D convolution, the proposed model enhances performance. To validate the effectiveness of the Partition-Time Masking data augmentation method, experiments were conducted on the LRW video dataset using the DC-TCN model, achieving a peak accuracy of 92.15%. The ST3D model was validated on the LRW and LRW1000 video datasets, achieving a maximum accuracy of 56.1% on the LRW1000 dataset and 91.8% on the LRW dataset, outperforming current mainstream lip reading models and demonstrating superior performance on challenging easily confused samples.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China.
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China.
- The Department of Sports Science, College of Education, Zhejiang University, Hangzhou 310027, China.
ORCID-Profile: Xiangliang Zhang, Xiangzhi Liu, Yu Gu, Tong Li, Jibin Yin, Tao Liu
Förderung
- NSFC
- National Key Laboratory of Transient Impact
- A Project Supported by Scientific Research Fund of Zhejiang University
- Zhejiang Provincial Natural Science Foundation of China
- Special Instrument Development and Cultivation Project of Zhejiang University
Zitieren
APA
Zhang, X., Hu, Y., Liu, X., Gu, Y., Li, T., Yin, J., & Liu, T. (2025). A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model. Sensors. https://doi.org/10.3390/s25082366
Vancouver
Zhang X, Hu Y, Liu X, Gu Y, Li T, Yin J, et al. A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model. Sensors. 2025. doi:10.3390/s25082366
BibTeX
@article{Zhang2025,
author = {Zhang, Xiangliang and Hu, Yu and Liu, Xiangzhi and Gu, Yu and Li, Tong and Yin, Jibin and Liu, Tao},
title = {A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model},
journal = {Sensors},
year = {2025},
doi = {10.3390/s25082366}
}
EUVIMED