EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model

Xiangliang Zhang iD, Yu Hu, Xiangzhi Liu iD, Yu Gu iD, Tong Li iD, Jibin Yin iD, Tao Liu iD

Sensors · 2025

DOI: 10.3390/s25082366 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2025. Verfasst von Xiangliang Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: The State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China. Hauptthemen laut MEDLINE: Speech Recognition Software, Speech. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSFC, National Key Laboratory of Transient Impact, A Project Supported by Scientific Research Fund of Zhejiang University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Visual speech recognition is a technology that relies on visual information, offering unique advantages in noisy environments or when communicating with individuals with speech impairments. However, this technology still faces challenges, such as limited generalization ability due to different speech habits, high recognition error rates caused by confusable phonemes, and difficulties adapting to complex lighting conditions and facial occlusions. This paper proposes a lip reading data augmentation method—Partition-Time Masking (PTM)—to address these challenges and improve lip reading models’ performance and generalization ability. Applying nonlinear transformations to the training data enhances the model’s generalization ability when handling diverse speakers and environmental conditions. A lip-reading recognition model architecture, Swin Transformer and 3D Convolution (ST3D), was designed to overcome the limitations of traditional lip-reading models that use ResNet-based front-end feature extraction networks. By adopting a strategy that combines Swin Transformer and 3D convolution, the proposed model enhances performance. To validate the effectiveness of the Partition-Time Masking data augmentation method, experiments were conducted on the LRW video dataset using the DC-TCN model, achieving a peak accuracy of 92.15%. The ST3D model was validated on the LRW and LRW1000 video datasets, achieving a maximum accuracy of 56.1% on the LRW1000 dataset and 91.8% on the LRW dataset, outperforming current mainstream lip reading models and demonstrating superior performance on challenging easily confused samples.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Speech Recognition SoftwareSpeechHumansAlgorithmsNeural Networks, Computer

Stichwörter der Autor:innen

data augmentationdeep learninghuman–computer interactionlip readingrecognition algorithms

Autor:innen und Institutionen

  • The State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China.
  • Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China.
  • The Department of Sports Science, College of Education, Zhejiang University, Hangzhou 310027, China.

ORCID-Profile: Xiangliang Zhang, Xiangzhi Liu, Yu Gu, Tong Li, Jibin Yin, Tao Liu

Förderung

  • NSFC
  • National Key Laboratory of Transient Impact
  • A Project Supported by Scientific Research Fund of Zhejiang University
  • Zhejiang Provincial Natural Science Foundation of China
  • Special Instrument Development and Cultivation Project of Zhejiang University

Zitieren

APA

Zhang, X., Hu, Y., Liu, X., Gu, Y., Li, T., Yin, J., & Liu, T. (2025). A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model. Sensors. https://doi.org/10.3390/s25082366

Vancouver

Zhang X, Hu Y, Liu X, Gu Y, Li T, Yin J, et al. A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model. Sensors. 2025. doi:10.3390/s25082366

BibTeX
@article{Zhang2025,
  author = {Zhang, Xiangliang and Hu, Yu and Liu, Xiangzhi and Gu, Yu and Li, Tong and Yin, Jibin and Liu, Tao},
  title = {A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional Model},
  journal = {Sensors},
  year = {2025},
  doi = {10.3390/s25082366}
}
Kostenlos starten →