EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

AFT-SAM: Adaptive Fusion Transformer with a Sparse Attention Mechanism for Audio–Visual Speech Recognition

Na Che, Yiming Zhu, Haiyan Wang, Xianwei Zeng, Qinsheng Du

Applied Sciences · 29.12.2024 · 15 (1) · 199

DOI: 10.3390/app15010199 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 29.12.2024. Verfasst von Na Che und 4 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science and Technology Research Project of the Education Department of Jilin Province, Key Project of the Jilin Special Education Society, Jilin Province Science and Technology Development Plan Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aiming at the problems of serious information redundancy, complex inter-modal information interaction, and difficult multimodal fusion faced by the audio–visual speech recognition system when dealing with complex multimodal information, this paper proposes an adaptive fusion transformer algorithm (AFT-SAM) based on a sparse attention mechanism. The algorithm adopts the sparse attention mechanism in the feature-encoding process to reduce excessive attention to non-important regions and dynamically adjusts the attention weights through adaptive fusion to capture and integrate the multimodal information more effectively and reduce the impact of redundant information on the model performance. Experiments are conducted on the audio–visual speech recognition dataset LRS2 and compared with other algorithms, and the experimental results show that the proposed algorithm in this paper has significantly lower WERs in the audio-only, visual-only, and audio–visual bimodal cases.

Förderung

  • Science and Technology Research Project of the Education Department of Jilin Province
  • Key Project of the Jilin Special Education Society
  • Jilin Province Science and Technology Development Plan Project
  • Changchun Science and Technology Development Plan Project
  • Science and Technology Development Plan Project of the Jilin Provincial Science and Technology Department

Zitieren

APA

Che, N., Zhu, Y., Wang, H., Zeng, X., & Du, Q. (2024). AFT-SAM: Adaptive Fusion Transformer with a Sparse Attention Mechanism for Audio–Visual Speech Recognition. Applied Sciences, 15(1), 199. https://doi.org/10.3390/app15010199

Vancouver

Che N, Zhu Y, Wang H, Zeng X, Du Q. AFT-SAM: Adaptive Fusion Transformer with a Sparse Attention Mechanism for Audio–Visual Speech Recognition. Applied Sciences. 2024;15(1):199. doi:10.3390/app15010199

BibTeX
@article{Che2024,
  author = {Che, Na and Zhu, Yiming and Wang, Haiyan and Zeng, Xianwei and Du, Qinsheng},
  title = {AFT-SAM: Adaptive Fusion Transformer with a Sparse Attention Mechanism for Audio–Visual Speech Recognition},
  journal = {Applied Sciences},
  year = {2024},
  volume = {15},
  number = {1},
  pages = {199},
  doi = {10.3390/app15010199}
}
Kostenlos starten →