Modality-agnostic decoding of vision and language from fMRI
eLife · 2026
DOI: 10.7554/elife.107933 ↗Kurz gefasst
Fachartikel in der Zeitschrift eLife, 2026. Verfasst von Mitja Nikolaus und 5 weiteren Autor:innen. Beteiligte Einrichtung: Université de Toulouse, CNRS, CerCo, Toulouse, France. Hauptthemen laut MEDLINE: Magnetic Resonance Imaging, Language, Brain, Visual Perception. Die Arbeit nennt 135 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Research Council, Agence Nationale de la Recherche.
Abstract (Originalsprache)
Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises six subjects watching both images and short text descriptions of such images, as well as the conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders and even outperform them when decoding captions and mental imagery. Furthermore, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Université de Toulouse, CNRS, CerCo, Toulouse, France.
- Torus AI, Toulouse, France.
- Université de Toulouse, IRIT, Toulouse, France.
ORCID-Profile: Mitja Nikolaus, Milad Mozafari, Rufin VanRullen
Förderung
- European Research Council
- Agence Nationale de la Recherche
Zitieren
APA
Nikolaus, M., Mozafari, M., Berry, I., Asher, N., Reddy, L., & VanRullen, R. (2026). Modality-agnostic decoding of vision and language from fMRI. eLife. https://doi.org/10.7554/elife.107933
Vancouver
Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. Modality-agnostic decoding of vision and language from fMRI. eLife. 2026. doi:10.7554/elife.107933
BibTeX
@article{Nikolaus2026,
author = {Nikolaus, Mitja and Mozafari, Milad and Berry, Isabelle and Asher, Nicholas and Reddy, Leila and VanRullen, Rufin},
title = {Modality-agnostic decoding of vision and language from fMRI},
journal = {eLife},
year = {2026},
doi = {10.7554/elife.107933}
}
EUVIMED