EUVIMEDEUVIMED entdecken
Crossref

Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning

Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, Eng Siong Chng

Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (11) · 12607-12615

DOI: 10.1609/aaai.v37i11.26484 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Chen Chen und 5 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio modality as it is much easier to recognize than video modality in clean conditions. As a result, the AVSR model underestimates the importance of visual stream in face of noise corruption. To this end, we leverage visual modality-specific representations to provide stable complementary information for the AVSR task. Specifically, we propose a reinforcement learning (RL) based framework called MSRL, where the agent dynamically harmonizes modality-invariant and modality-specific representations in the auto-regressive decoding process. We customize a reward function directly related to task-specific metrics (i.e., word error rate), which encourages the MSRL to effectively explore the optimal integration strategy. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions. Furthermore, we demonstrate the better generality of MSRL system than other baselines when test set contains unseen noises.

Zitieren

APA

Chen, C., Hu, Y., Zhang, Q., Zou, H., Zhu, B., & Chng, E. S. (2023). Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 12607-12615. https://doi.org/10.1609/aaai.v37i11.26484

Vancouver

Chen C, Hu Y, Zhang Q, Zou H, Zhu B, Chng ES. Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(11):12607-12615. doi:10.1609/aaai.v37i11.26484

BibTeX
@article{Chen2023,
  author = {Chen, Chen and Hu, Yuchen and Zhang, Qiang and Zou, Heqing and Zhu, Beier and Chng, Eng Siong},
  title = {Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2023},
  volume = {37},
  number = {11},
  pages = {12607--12615},
  doi = {10.1609/aaai.v37i11.26484}
}
Kostenlos starten →