Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models
PLOS Computational Biology · 30.10.2026 · 22 (10) · e1014906
DOI: 10.1371/journal.pcbi.1014906 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 30.10.2026. Verfasst von Riki Shimizu und 3 weiteren Autor:innen. Beteiligte Einrichtung: Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America. Hauptthemen laut MEDLINE: Brain, Speech Perception, Speech, Models, Neurological. Die Arbeit nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, National Institutes of Health.
Abstract (Originalsprache)
Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America.
- Microsoft Research, Redmond, Washington, United States of America.
ORCID-Profile: Riki Shimizu
Förderung
- NIDCD NIH HHS
- National Institutes of Health
Zitieren
APA
Shimizu, R., Antonello, R. J., Singh, C., & Mesgarani, N. (2026). Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models. PLOS Computational Biology, 22(10), e1014906. https://doi.org/10.1371/journal.pcbi.1014906
Vancouver
Shimizu R, Antonello RJ, Singh C, Mesgarani N. Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models. PLOS Computational Biology. 2026;22(10):e1014906. doi:10.1371/journal.pcbi.1014906
BibTeX
@article{Shimizu2026,
author = {Shimizu, Riki and Antonello, Richard J and Singh, Chandan and Mesgarani, Nima},
title = {Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models},
journal = {PLOS Computational Biology},
year = {2026},
volume = {22},
number = {10},
pages = {e1014906},
doi = {10.1371/journal.pcbi.1014906}
}
EUVIMED