EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation

Jong-Yun Park iD, Mitsuaki Tsukamoto, Misato Tanaka, Yukiyasu Kamitani iD

PLOS Biology · 2025

DOI: 10.1371/journal.pbio.3003293 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Biology, 2025. Verfasst von Jong-Yun Park und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Kyoto, Japan. Hauptthemen laut MEDLINE: Auditory Perception, Neural Networks, Computer, Neuroimaging, Sound. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science (JSPS), New Energy and Industrial Technology Development Organization, Japan Science and Technology Corporation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reconstruction of perceptual experiences from brain activity offers a unique window into how population neural responses represent sensory information. Although decoding visual content from functional MRI (fMRI) has seen significant success, reconstructing arbitrary sounds remains challenging due to the fine temporal structure of auditory signals and the coarse temporal resolution of fMRI. Drawing on the hierarchical auditory features of deep neural networks (DNNs) with progressively larger time windows and their neural activity correspondence, we introduce a method for sound reconstruction that integrates brain decoding of DNN features and an audio-generative model. DNN features decoded from auditory cortical activity outperformed spectrotemporal and modulation-based features, enabling perceptually plausible reconstructions across diverse sound categories. Behavioral evaluations and objective measures confirmed that these reconstructions preserved short-term spectral and perceptual properties, capturing the characteristic timbre of speech, animal calls, and musical instruments, while the reconstructed sounds did not reproduce longer temporal sequences with fidelity. Leave-category-out analyses indicated that the method generalizes across sound categories. Reconstructions at higher DNN layers and from early auditory regions revealed distinct contributions to decoding performance. Applying the model to a selective auditory attention (“cocktail party”) task further showed that reconstructions reflected the attended sound more strongly than the unattended one in some of the subjects. Despite its inability to reconstruct exact temporal sequences, which may reflect the limited temporal resolution of fMRI, our framework demonstrates the feasibility of mapping brain activity to auditory experiences—a step toward more comprehensive understanding and reconstruction of internal auditory representations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Auditory PerceptionNeural Networks, ComputerNeuroimagingSoundHumansMagnetic Resonance ImagingAuditory CortexFemaleMaleAcoustic StimulationAdultYoung AdultBrain MappingBrain

Autor:innen und Institutionen

  • Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Kyoto, Japan.
  • Department of Neuroinformatics, ATR Computational Neuroscience Laboratories, Kyoto, Japan.

ORCID-Profile: Jong-Yun Park, Yukiyasu Kamitani

Förderung

  • Japan Society for the Promotion of Science (JSPS)
  • New Energy and Industrial Technology Development Organization
  • Japan Science and Technology Corporation

Zitieren

APA

Park, J. Y., Tsukamoto, M., Tanaka, M., & Kamitani, Y. (2025). Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation. PLOS Biology. https://doi.org/10.1371/journal.pbio.3003293

Vancouver

Park JY, Tsukamoto M, Tanaka M, Kamitani Y. Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation. PLOS Biology. 2025. doi:10.1371/journal.pbio.3003293

BibTeX
@article{Park2025,
  author = {Park, Jong-Yun and Tsukamoto, Mitsuaki and Tanaka, Misato and Kamitani, Yukiyasu},
  title = {Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation},
  journal = {PLOS Biology},
  year = {2025},
  doi = {10.1371/journal.pbio.3003293}
}
Infokarte | EUVIMED

Artikeldaten derzeit nicht verfügbar.

Recherche