EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Contrastive learning to fine-tune feature extraction models for the visual cortex

Alex Mulrooney iD, Zhi Li, Austin J Brockmeier

PLOS Computational Biology · 17.08.2026 · 22 (8) · e1014656

DOI: 10.1371/journal.pcbi.1014656 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 17.08.2026. Verfasst von Alex Mulrooney und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical and Computer Engineering, University of Delaware, Newark, Delaware, United States of America. Hauptthemen laut MEDLINE: Visual Cortex, Models, Neurological. Die Arbeit nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Delaware General University Research fund, Office of Naval Research, University of Delaware Department of Electrical and Computer Engineering First Year Fellowship.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Predicting the neural response to natural images in the visual cortex requires extracting relevant features from the images and relating those feature to the observed responses. In this work, we optimize the feature extraction in order to maximize the information shared between the image features and the neural response across voxels in a given region of interest (ROI) extracted from the BOLD signal measured by functional magnetic resonance imaging (fMRI). We adapt contrastive learning (CL) to fine-tune a convolutional neural network, which was pretrained for image classification, such that a mapping of a given image’s features are more similar to the corresponding fMRI response than to the responses to other images. We exploit the Natural Scenes Dataset as organized for the Algonauts Project, which contains the high-resolution fMRI responses of eight subjects to tens of thousands of naturalistic images. We show that CL fine-tuning creates feature extraction models that enable higher encoding accuracy in both early and higher visual ROIs as compared to the features from the pretrained network. Quantitatively, the performance is similar to a baseline approach that directly uses a regression loss at the output of the network to tune it for fMRI response encoding. We investigate inter-subject transfer of the CL fine-tuned models, including subjects from the Natural Object Dataset, another lower-resolution dataset with 9 subjects. We also pool subjects for fine-tuning, which further improves encoding performance in early ROIs. Finally, we examine the performance of the fine-tuned models on common image classification tasks, explore the landscape of ROI-specific models by applying dimensionality reduction on the Bhattacharya dissimilarity matrix created using the predictions on those tasks, and show that these landscapes match those based on representational similarity analysis. Finally, we generate images via Stable Diffusion based on vector-space prompts created by aligning the CL-tuned models embeddings for different ROIs, showing that generated images have similar embeddings to the original but that estimates of the intrinsic dimension are lower for generated versus original representations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Visual CortexModels, NeurologicalHumansMagnetic Resonance ImagingConvolutional Neural NetworksBrain MappingComputational BiologyImage Processing, Computer-AssistedAlgorithms

Autor:innen und Institutionen

  • Department of Electrical and Computer Engineering, University of Delaware, Newark, Delaware, United States of America.
  • Department of Computer and Information Sciences, University of Delaware, Newark, Delaware, United States of America.

ORCID-Profile: Alex Mulrooney

Förderung

  • University of Delaware General University Research fund
  • Office of Naval Research
  • University of Delaware Department of Electrical and Computer Engineering First Year Fellowship
  • University of Delaware Research Foundation
  • University of Delaware Electrical and Computer Engineering Department
  • University of Delaware College of Engineering

Zitieren

APA

Mulrooney, A., Li, Z., & Brockmeier, A. J. (2026). Contrastive learning to fine-tune feature extraction models for the visual cortex. PLOS Computational Biology, 22(8), e1014656. https://doi.org/10.1371/journal.pcbi.1014656

Vancouver

Mulrooney A, Li Z, Brockmeier AJ. Contrastive learning to fine-tune feature extraction models for the visual cortex. PLOS Computational Biology. 2026;22(8):e1014656. doi:10.1371/journal.pcbi.1014656

BibTeX
@article{Mulrooney2026,
  author = {Mulrooney, Alex and Li, Zhi and Brockmeier, Austin J},
  title = {Contrastive learning to fine-tune feature extraction models for the visual cortex},
  journal = {PLOS Computational Biology},
  year = {2026},
  volume = {22},
  number = {8},
  pages = {e1014656},
  doi = {10.1371/journal.pcbi.1014656}
}
Kostenlos starten →