EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation

Richard Lee Lai, Jen-Cheng Hou, I-Chun Chern, Kuo-Hsuan Hung, Yi-Ting Chen, Mandar Gogate, Tughrul Arslan, Amir Hussain, Chii-Wann Lin, Yu Tsao

IEEE Transactions on Biomedical Engineering · 2026-04 · 73 (4) · 1561-1572

DOI: 10.1109/tbme.2025.3610284 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Biomedical Engineering, 2026-04. Verfasst von Richard Lee Lai und 9 weiteren Autor:innen. Hauptthemen laut MEDLINE: Cochlear Implants, Speech Intelligibility, Speech Perception, Supervised Machine Learning. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 115 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

OBJECTIVE: Individuals with hearing impairments face challenges in their ability to comprehend speech, particularly in noisy environments. This study explores the effectiveness of audio-visual speech enhancement (AVSE) in improving the intelligibility of vocoded speech in cochlear implant (CI) simulations. METHODS: We propose a speech enhancement framework called Self-Supervised Learning-based AVSE (SSL-AVSE), which uses visual cues such as lip and mouth movements along with corresponding speech. Features are extracted using the AV-HuBERT model and refined through a bidirectional LSTM. Experiments were conducted using the Taiwan Mandarin speech with video (TMSV) dataset. RESULTS: Objective evaluations showed improvements in PESQ from 1.43 to 1.67 and in STOI from 0.70 to 0.74. NCM scores increased by up to 87.2% over the noisy baseline. Subjective listening tests further demonstrated maximum gains of 45.2% in speech quality and 51.9% in word intelligibility. CONCLUSION: SSL-AVSE consistently outperforms audio-only speech enhancement (AOSE) and conventional AVSE baselines. Listening tests with statistically significant results confirm its effectiveness. In addition to its strong performance, SSL-AVSE demonstrates cross-lingual generalization: although it was pretrained on English data, it performs effectively on Mandarin speech. This finding highlights the robustness of the features extracted by a pretrained foundation model and their applicability across languages. SIGNIFICANCE: To the best of our knowledge, no prior work has explored the application of AVSE to CI simulations. This study provides the first evidence that incorporating visual information can significantly improve the intelligibility of vocoded speech in CI scenarios.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Cochlear ImplantsSpeech IntelligibilitySpeech PerceptionSupervised Machine LearningSignal Processing, Computer-AssistedHumansMaleFemaleAdult

Zitieren

APA

Lai, R. L., Hou, J. C., Chern, I. C., Hung, K. H., Chen, Y. T., Gogate, M., Arslan, T., Hussain, A., Lin, C. W., & Tsao, Y. (2026). Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation. IEEE Transactions on Biomedical Engineering, 73(4), 1561-1572. https://doi.org/10.1109/tbme.2025.3610284

Vancouver

Lai RL, Hou JC, Chern IC, Hung KH, Chen YT, Gogate M, et al. Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation. IEEE Transactions on Biomedical Engineering. 2026;73(4):1561-1572. doi:10.1109/tbme.2025.3610284

BibTeX
@article{Lai2026,
  author = {Lai, Richard Lee and Hou, Jen-Cheng and Chern, I-Chun and Hung, Kuo-Hsuan and Chen, Yi-Ting and Gogate, Mandar and Arslan, Tughrul and Hussain, Amir and Lin, Chii-Wann and Tsao, Yu},
  title = {Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation},
  journal = {IEEE Transactions on Biomedical Engineering},
  year = {2026},
  volume = {73},
  number = {4},
  pages = {1561--1572},
  doi = {10.1109/tbme.2025.3610284}
}
Kostenlos starten →