Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 31247-31255
DOI: 10.1609/aaai.v40i37.40387 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Jinhong Jeong und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Sound symbolism is a linguistic concept that refers to non-arbitrary associations between phonetic forms and their meanings. We suggest that this can be a compelling probe into how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages. We investigate MLLMs' performance on phonetic iconicity across textual (orthographic and IPA) and auditory forms of inputs with up to 25 semantic dimensions (e.g., sharp vs. round), observing models' layer-wise information processing by measuring phoneme-level attention fraction scores. To this end, we present LEX-ICON, an extensive mimetic word dataset consisting of 8,052 words from four natural languages (English, French, Japanese, and Korean) and 2,930 systematically constructed pseudo-words, annotated with semantic features applied across both text and audio modalities. Our key findings demonstrate (1) MLLMs' phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic attention patterns that highlight models' focus on iconic phonemes. These results bridge domains of artificial intelligence and cognitive linguistics, providing the first large-scale, quantitative analyses of phonetic iconicity in terms of MLLMs' interpretability.
Zitieren
APA
Jeong, J., Lee, S., Lee, J., Han, S., & Yu, Y. (2026). Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 31247-31255. https://doi.org/10.1609/aaai.v40i37.40387
Vancouver
Jeong J, Lee S, Lee J, Han S, Yu Y. Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):31247-31255. doi:10.1609/aaai.v40i37.40387
BibTeX
@article{Jeong2026,
author = {Jeong, Jinhong and Lee, Sunghyun and Lee, Jaeyoung and Han, Seonah and Yu, Youngjae},
title = {Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {37},
pages = {31247--31255},
doi = {10.1609/aaai.v40i37.40387}
}
EUVIMED