EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Probing the link between vision and language in material perception using psychophysics and unsupervised learning

Chenxi Liao iD, Masataka Sawayama iD, Bei Xiao

PLOS Computational Biology · 03.10.2024 · 20 (10) · e1012481

DOI: 10.1371/journal.pcbi.1012481 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 03.10.2024. Verfasst von Chenxi Liao und 2 weiteren Autor:innen. Beteiligte Einrichtung: American University, Department of Neuroscience, Washington DC, United States of America. Hauptthemen laut MEDLINE: Unsupervised Machine Learning, Language, Visual Perception, Psychophysics. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NEI NIH HHS, National Eye Institute.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to describe what we see and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression to understand how visual features relate to semantic representations in human cognition. We use deep generative models to generate images of realistic materials. Interpolating between the generative models enables us to systematically create material appearances in both well-defined and ambiguous categories. Using these stimuli, we compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among ambiguous materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual nuances of material appearances. Analyzing the image representation of materials obtained from various pre-trained deep neural networks, we find that similarity structures in human visual judgments align more closely with those of the vision-language models than purely vision-based models. Our work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Unsupervised Machine LearningLanguageVisual PerceptionPsychophysicsHumansComputational BiologySemanticsVision, OcularJudgmentDeep LearningNeural Networks, ComputerCognition

Autor:innen und Institutionen

  • American University, Department of Neuroscience, Washington DC, United States of America.
  • The University of Tokyo, Graduate School of Information Science and Technology, Tokyo, Japan.
  • American University, Department of Computer Science, Washington DC, United States of America.

ORCID-Profile: Chenxi Liao, Masataka Sawayama

Förderung

  • NEI NIH HHS
  • National Eye Institute

Zitieren

APA

Liao, C., Sawayama, M., & Xiao, B. (2024). Probing the link between vision and language in material perception using psychophysics and unsupervised learning. PLOS Computational Biology, 20(10), e1012481. https://doi.org/10.1371/journal.pcbi.1012481

Vancouver

Liao C, Sawayama M, Xiao B. Probing the link between vision and language in material perception using psychophysics and unsupervised learning. PLOS Computational Biology. 2024;20(10):e1012481. doi:10.1371/journal.pcbi.1012481

BibTeX
@article{Liao2024,
  author = {Liao, Chenxi and Sawayama, Masataka and Xiao, Bei},
  title = {Probing the link between vision and language in material perception using psychophysics and unsupervised learning},
  journal = {PLOS Computational Biology},
  year = {2024},
  volume = {20},
  number = {10},
  pages = {e1012481},
  doi = {10.1371/journal.pcbi.1012481}
}
Kostenlos starten →