EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models

Riki Shimizu iD, Richard J Antonello, Chandan Singh, Nima Mesgarani

PLOS Computational Biology · 30.10.2026 · 22 (10) · e1014906

DOI: 10.1371/journal.pcbi.1014906 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 30.10.2026. Verfasst von Riki Shimizu und 3 weiteren Autor:innen. Beteiligte Einrichtung: Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America. Hauptthemen laut MEDLINE: Brain, Speech Perception, Speech, Models, Neurological. Die Arbeit nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BrainSpeech PerceptionSpeechModels, NeurologicalHumansElectrocorticographyComputational BiologySemanticsComputer Simulation

Autor:innen und Institutionen

  • Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America.
  • Microsoft Research, Redmond, Washington, United States of America.

ORCID-Profile: Riki Shimizu

Förderung

  • NIDCD NIH HHS
  • National Institutes of Health

Zitieren

APA

Shimizu, R., Antonello, R. J., Singh, C., & Mesgarani, N. (2026). Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models. PLOS Computational Biology, 22(10), e1014906. https://doi.org/10.1371/journal.pcbi.1014906

Vancouver

Shimizu R, Antonello RJ, Singh C, Mesgarani N. Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models. PLOS Computational Biology. 2026;22(10):e1014906. doi:10.1371/journal.pcbi.1014906

BibTeX
@article{Shimizu2026,
  author = {Shimizu, Riki and Antonello, Richard J and Singh, Chandan and Mesgarani, Nima},
  title = {Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models},
  journal = {PLOS Computational Biology},
  year = {2026},
  volume = {22},
  number = {10},
  pages = {e1014906},
  doi = {10.1371/journal.pcbi.1014906}
}
Kostenlos starten →