EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Modality-agnostic decoding of vision and language from fMRI

Mitja Nikolaus iD, Milad Mozafari iD, Isabelle Berry, Nicholas Asher, Leila Reddy, Rufin VanRullen iD

eLife · 2026

DOI: 10.7554/elife.107933 ↗

Kurz gefasst

Fachartikel in der Zeitschrift eLife, 2026. Verfasst von Mitja Nikolaus und 5 weiteren Autor:innen. Beteiligte Einrichtung: Université de Toulouse, CNRS, CerCo, Toulouse, France. Hauptthemen laut MEDLINE: Magnetic Resonance Imaging, Language, Brain, Visual Perception. Die Arbeit nennt 135 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Research Council, Agence Nationale de la Recherche.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises six subjects watching both images and short text descriptions of such images, as well as the conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders and even outperform them when decoding captions and mental imagery. Furthermore, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Magnetic Resonance ImagingLanguageBrainVisual PerceptionVision, OcularHumansBrain MappingPhotic Stimulation

Stichwörter der Autor:innen

decodingfMRIhumanimagerymodality-invariantneurosciencesearchlight

Autor:innen und Institutionen

  • Université de Toulouse, CNRS, CerCo, Toulouse, France.
  • Torus AI, Toulouse, France.
  • Université de Toulouse, IRIT, Toulouse, France.

ORCID-Profile: Mitja Nikolaus, Milad Mozafari, Rufin VanRullen

Förderung

  • European Research Council
  • Agence Nationale de la Recherche

Zitieren

APA

Nikolaus, M., Mozafari, M., Berry, I., Asher, N., Reddy, L., & VanRullen, R. (2026). Modality-agnostic decoding of vision and language from fMRI. eLife. https://doi.org/10.7554/elife.107933

Vancouver

Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. Modality-agnostic decoding of vision and language from fMRI. eLife. 2026. doi:10.7554/elife.107933

BibTeX
@article{Nikolaus2026,
  author = {Nikolaus, Mitja and Mozafari, Milad and Berry, Isabelle and Asher, Nicholas and Reddy, Leila and VanRullen, Rufin},
  title = {Modality-agnostic decoding of vision and language from fMRI},
  journal = {eLife},
  year = {2026},
  doi = {10.7554/elife.107933}
}
Kostenlos starten →