EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset

James Fodor, Simon De Deyne, Shinsuke Suzuki

Computational Linguistics · 15.03.2025 · 51 (1) · 139-190

DOI: 10.1162/coli_a_00536 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computational Linguistics, online erschienen am 15.03.2025. Verfasst von James Fodor und 2 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 170 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

One of the major outstanding questions in computational semantics is how humans integrate the meaning of individual words into a sentence in a way that enables understanding of complex and novel combinations of words, a phenomenon known as compositionality. Many approaches to modeling the process of compositionality can be classified as either “vector-based” models, in which the meaning of a sentence is represented as a vector of numbers, or “syntax-based” models, in which the meaning of a sentence is represented as a structured tree of labeled components. A major barrier in assessing and comparing these contrasting approaches is the lack of large, relevant datasets for model comparison. This article aims to address this gap by introducing a new dataset, STS3k, which consists of 2,800 pairs of sentences rated for semantic similarity by human participants. The sentence pairs have been selected to systematically vary different combinations of words, providing a rigorous test and enabling a clearer picture of the comparative strengths and weaknesses of vector-based and syntax-based methods. Our results show that when tested on the new STS3k dataset, state-of-the-art transformers poorly capture the pattern of human semantic similarity judgments, while even simple methods for combining syntax- and vector-based components into a novel hybrid model yield substantial improvements. We further show that this improvement is due to the ability of the hybrid model to replicate human sensitivity to specific changes in sentence structure. Our findings provide evidence for the value of integrating multiple methods to better reflect the way in which humans mentally represent compositional meaning.

Zitieren

APA

Fodor, J., Deyne, S. D., & Suzuki, S. (2025). Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset. Computational Linguistics, 51(1), 139-190. https://doi.org/10.1162/coli_a_00536

Vancouver

Fodor J, Deyne SD, Suzuki S. Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset. Computational Linguistics. 2025;51(1):139-190. doi:10.1162/coli_a_00536

BibTeX
@article{Fodor2025,
  author = {Fodor, James and Deyne, Simon De and Suzuki, Shinsuke},
  title = {Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset},
  journal = {Computational Linguistics},
  year = {2025},
  volume = {51},
  number = {1},
  pages = {139--190},
  doi = {10.1162/coli_a_00536}
}
Kostenlos starten →