Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation
PLOS Computational Biology · 11.11.2024 · 20 (11) · e1012537
DOI: 10.1371/journal.pcbi.1012537 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 11.11.2024. Verfasst von Andrew J Anderson und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology. Medical College of Wisconsin, Milwaukee, Wisconsin United States of America. Hauptthemen laut MEDLINE: Deep Learning, Electroencephalography, Speech Perception, Attention. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Del Monte Institute Pilot Project Program, Advancing a Healthier Wisconsin Endowment (AJA), Australian Research Council.
Abstract (Originalsprache)
To transform continuous speech into words, the human brain must resolve variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke context invariant speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such models may not capture the complete picture because they do not model the brain mechanism that categorizes sounds and consequently may overlook associated neural representations. By providing end-to-end accounts of speech-to-text transformation, new deep-learning systems could enable more complete brain models. We model EEG recordings of audiobook comprehension with the deep-learning speech recognition system Whisper. We find that (1) Whisper provides a self-contained EEG model of an intermediary representational stage that reflects elements of prelexical and lexical representation and prediction; (2) EEG modeling is more accurate when informed by 5-10s of speech context, which traditional context invariant categorical models do not encode; (3) Deep Whisper layers encoding linguistic structure were more accurate EEG models of selectively attended speech in two-speaker “cocktail party” listening conditions than early layers encoding acoustics. No such layer depth advantage was observed for unattended speech, consistent with a more superficial level of linguistic processing in the brain.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Neurology. Medical College of Wisconsin, Milwaukee, Wisconsin United States of America.
- Department of Biomedical Engineering. Medical College of Wisconsin. Milwaukee, Wisconsin United States of America.
- Department of Neurosurgery. Medical College of Wisconsin. Milwaukee, Wisconsin United States of America.
- Department of Neuroscience and Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York, United States of America.
- Western Sydney University, The MARCS Institute for Brain, Behaviour and Development, Westmead Innovation Quarter, Westmead, New South Wales, Australia.
- Department of Biomedical Engineering, University of Rochester, Rochester, New York, United States of America.
ORCID-Profile: Andrew J Anderson, Chris Davis, Edmund C Lalor
Förderung
- Del Monte Institute Pilot Project Program
- Advancing a Healthier Wisconsin Endowment (AJA)
- Australian Research Council
- National Science Foundation
- Del Monte Institute
Zitieren
APA
Anderson, A. J., Davis, C., & Lalor, E. C. (2024). Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation. PLOS Computational Biology, 20(11), e1012537. https://doi.org/10.1371/journal.pcbi.1012537
Vancouver
Anderson AJ, Davis C, Lalor EC. Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation. PLOS Computational Biology. 2024;20(11):e1012537. doi:10.1371/journal.pcbi.1012537
BibTeX
@article{Anderson2024,
author = {Anderson, Andrew J and Davis, Chris and Lalor, Edmund C},
title = {Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation},
journal = {PLOS Computational Biology},
year = {2024},
volume = {20},
number = {11},
pages = {e1012537},
doi = {10.1371/journal.pcbi.1012537}
}
EUVIMED