EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Julia Berezutskaya iD, Zachary V Freudenburg iD, Mariska J Vansteensel iD, Erik J Aarnoutse iD, Nick F Ramsey iD, Marcel A J van Gerven iD

Journal of Neural Engineering · 01.10.2023 · 20 (5) · 056010

DOI: 10.1088/1741-2552/ace8be ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Neural Engineering, online erschienen am 20.09.2023. Verfasst von Julia Berezutskaya und 5 weiteren Autor:innen. Beteiligte Einrichtung: Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands. Hauptthemen laut MEDLINE: Deep Learning, Brain-Computer Interfaces, Sensorimotor Cortex. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 83 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, Nederlandse Organisatie voor Wetenschappelijk Onderzoek, National Institute on Deafness and Other Communication Disorders.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective. Development of brain–computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field. Approach. In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task. Main results. We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%–100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech. Significance. These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Deep LearningBrain-Computer InterfacesSensorimotor CortexHumansSpeechCommunicationElectrocorticography

Stichwörter der Autor:innen

audio reconstructionbrainbrain–computer interfacesdeep neural networkselectrocorticographyneural decodingspeech

Autor:innen und Institutionen

  • Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands.
  • Donders Center for Brain, Cognition and Behaviour, Nijmegen 6525 GD, The Netherlands.

ORCID-Profile: Julia Berezutskaya, Zachary V Freudenburg, Mariska J Vansteensel, Erik J Aarnoutse, Nick F Ramsey, Marcel A J van Gerven

Förderung

  • NIDCD NIH HHS
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek
  • National Institute on Deafness and Other Communication Disorders
  • European Research Council

Zitieren

APA

Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2023). Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models. Journal of Neural Engineering, 20(5), 056010. https://doi.org/10.1088/1741-2552/ace8be

Vancouver

Berezutskaya J, Freudenburg ZV, Vansteensel MJ, Aarnoutse EJ, Ramsey NF, van Gerven MAJ. Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models. Journal of Neural Engineering. 2023;20(5):056010. doi:10.1088/1741-2552/ace8be

BibTeX
@article{Berezutskaya2023,
  author = {Berezutskaya, Julia and Freudenburg, Zachary V and Vansteensel, Mariska J and Aarnoutse, Erik J and Ramsey, Nick F and van Gerven, Marcel A J},
  title = {Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models},
  journal = {Journal of Neural Engineering},
  year = {2023},
  volume = {20},
  number = {5},
  pages = {056010},
  doi = {10.1088/1741-2552/ace8be}
}
Kostenlos starten →