EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation

Andrew J Anderson iD, Chris Davis iD, Edmund C Lalor iD

PLOS Computational Biology · 11.11.2024 · 20 (11) · e1012537

DOI: 10.1371/journal.pcbi.1012537 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 11.11.2024. Verfasst von Andrew J Anderson und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology. Medical College of Wisconsin, Milwaukee, Wisconsin United States of America. Hauptthemen laut MEDLINE: Deep Learning, Electroencephalography, Speech Perception, Attention. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Del Monte Institute Pilot Project Program, Advancing a Healthier Wisconsin Endowment (AJA), Australian Research Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To transform continuous speech into words, the human brain must resolve variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke context invariant speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such models may not capture the complete picture because they do not model the brain mechanism that categorizes sounds and consequently may overlook associated neural representations. By providing end-to-end accounts of speech-to-text transformation, new deep-learning systems could enable more complete brain models. We model EEG recordings of audiobook comprehension with the deep-learning speech recognition system Whisper. We find that (1) Whisper provides a self-contained EEG model of an intermediary representational stage that reflects elements of prelexical and lexical representation and prediction; (2) EEG modeling is more accurate when informed by 5-10s of speech context, which traditional context invariant categorical models do not encode; (3) Deep Whisper layers encoding linguistic structure were more accurate EEG models of selectively attended speech in two-speaker “cocktail party” listening conditions than early layers encoding acoustics. No such layer depth advantage was observed for unattended speech, consistent with a more superficial level of linguistic processing in the brain.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Deep LearningElectroencephalographySpeech PerceptionAttentionLanguageHumansBrainSpeechMaleFemaleAdultComputational BiologyYoung AdultModels, Neurological

Autor:innen und Institutionen

  • Department of Neurology. Medical College of Wisconsin, Milwaukee, Wisconsin United States of America.
  • Department of Biomedical Engineering. Medical College of Wisconsin. Milwaukee, Wisconsin United States of America.
  • Department of Neurosurgery. Medical College of Wisconsin. Milwaukee, Wisconsin United States of America.
  • Department of Neuroscience and Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York, United States of America.
  • Western Sydney University, The MARCS Institute for Brain, Behaviour and Development, Westmead Innovation Quarter, Westmead, New South Wales, Australia.
  • Department of Biomedical Engineering, University of Rochester, Rochester, New York, United States of America.

ORCID-Profile: Andrew J Anderson, Chris Davis, Edmund C Lalor

Förderung

  • Del Monte Institute Pilot Project Program
  • Advancing a Healthier Wisconsin Endowment (AJA)
  • Australian Research Council
  • National Science Foundation
  • Del Monte Institute

Zitieren

APA

Anderson, A. J., Davis, C., & Lalor, E. C. (2024). Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation. PLOS Computational Biology, 20(11), e1012537. https://doi.org/10.1371/journal.pcbi.1012537

Vancouver

Anderson AJ, Davis C, Lalor EC. Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation. PLOS Computational Biology. 2024;20(11):e1012537. doi:10.1371/journal.pcbi.1012537

BibTeX
@article{Anderson2024,
  author = {Anderson, Andrew J and Davis, Chris and Lalor, Edmund C},
  title = {Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation},
  journal = {PLOS Computational Biology},
  year = {2024},
  volume = {20},
  number = {11},
  pages = {e1012537},
  doi = {10.1371/journal.pcbi.1012537}
}
Kostenlos starten →