Simultaneous speech and gesture decoding for multimodal communication in paralysis
Nature Neuroscience · 2026
DOI: 10.1038/s41593-026-02446-2 ↗Kurz gefasst
Fachartikel in der Zeitschrift Nature Neuroscience, 2026. Verfasst von Samantha C Brosler und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA. Hauptthemen laut MEDLINE: Gestures, Speech, Brain-Computer Interfaces, Paralysis. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, MEXT | Japan Science and Technology Agency (JST), U.S. Department of Health & Human Services | NIH | National Institute on Deafness and Other Communication Disorders (NIDCD).
Abstract (Originalsprache)
Stroke and neurodegenerative diseases can impair speech and nonverbal gestures, limiting natural communication. Brain–computer interfaces (BCIs) aim to restore these functions by translating neural activity into commands for external devices, although prior work has primarily focused on decoding speech or gestures in isolation. Here we show that neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech and gestures in people with paralysis. We first show that isolated upper-limb and orofacial movements can be reliably decoded among three participants. Using parallel speech and gesture decoders, we then enabled participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. Training models on both isolated and simultaneous data improved performance across behavioral contexts. These findings demonstrate that one cortical implant can support multi-effector control and provide a step toward BCIs that enable more natural communication for people with paralysis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA.
- University of California, Berkeley-University of California, San Francisco, Graduate Program in Bioengineering, Berkeley, CA, USA.
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA.
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA. edward.chang@ucsf.edu.
- Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA. edward.chang@ucsf.edu.
Förderung
- NIDCD NIH HHS
- MEXT | Japan Science and Technology Agency (JST)
- U.S. Department of Health & Human Services | NIH | National Institute on Deafness and Other Communication Disorders (NIDCD)
Zitieren
APA
Brosler, S. C., Liu, J. R., Silva, A. B., Hallinan, I. P., Kurtz-Miott, C. M., Dunkel Wilker, J. F. B., Tu-Chan, A., Ganguly, K., & Chang, E. F. (2026). Simultaneous speech and gesture decoding for multimodal communication in paralysis. Nature Neuroscience. https://doi.org/10.1038/s41593-026-02446-2
Vancouver
Brosler SC, Liu JR, Silva AB, Hallinan IP, Kurtz-Miott CM, Dunkel Wilker JFB, et al. Simultaneous speech and gesture decoding for multimodal communication in paralysis. Nature Neuroscience. 2026. doi:10.1038/s41593-026-02446-2
BibTeX
@article{Brosler2026,
author = {Brosler, Samantha C and Liu, Jessie R and Silva, Alexander B and Hallinan, Irina P and Kurtz-Miott, Cady M and Dunkel Wilker, Jonah F B and Tu-Chan, Adelyn and Ganguly, Karunesh and Chang, Edward F},
title = {Simultaneous speech and gesture decoding for multimodal communication in paralysis},
journal = {Nature Neuroscience},
year = {2026},
doi = {10.1038/s41593-026-02446-2}
}
EUVIMED