Derivational morphology reveals analogical generalization in large language models
Proceedings of the National Academy of Sciences · 2025
DOI: 10.1073/pnas.2423232122 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, 2025. Verfasst von Valentin Hofmann und 4 weiteren Autor:innen. Beteiligte Einrichtung: Allen Institute for AI, Seattle, WA 98103. Hauptthemen laut MEDLINE: Linguistics, Models, Theoretical. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 117 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UKRI | Engineering and Physical Sciences Research Council (EPSRC), German Academic Scholarship Foundation, German Academic Exchange Service (DAAD).
Abstract (Originalsprache)
What mechanisms underlie linguistic generalization in large language models (LLMs)? This question has attracted considerable attention, with most studies analyzing the extent to which the language skills of LLMs resemble rules. As of yet, it is not known whether linguistic generalization in LLMs could equally well be explained as the result of analogy. A key shortcoming of prior research is its focus on regular linguistic phenomena, for which rule-based and analogical approaches make the same predictions. Here, we instead examine derivational morphology, specifically English adjective nominalization, which displays notable variability. We introduce a method for investigating linguistic generalization in LLMs: Focusing on GPT-J, we fit cognitive models that instantiate rule-based and analogical learning to the LLM training data and compare their predictions on a set of nonce adjectives with those of the LLM, allowing us to draw direct conclusions regarding underlying mechanisms. As expected, rule-based and analogical models explain the predictions of GPT-J equally well for adjectives with regular nominalization patterns. However, for adjectives with variable nominalization patterns, the analogical model provides a much better match. Furthermore, GPT-J’s behavior is sensitive to the individual word frequencies, even for regular forms, a behavior that is consistent with an analogical account but not a rule-based one. These findings refute the hypothesis that GPT-J’s linguistic generalization on adjective nominalization involves rules, suggesting analogy as the underlying mechanism. Overall, our study suggests that analogical processes play a bigger role in the linguistic generalization of LLMs than previously thought.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Allen Institute for AI, Seattle, WA 98103.
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA 98195.
- Center for Information and Language Processing, Ludwig Maximilian University of Munich, Munich 80538, Germany.
- School of Computer Science, Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213.
- Oxford e-Research Centre, Department of Engineering Science, University of Oxford, Oxford, OX1 3QG, United Kingdom.
ORCID-Profile: Janet B Pierrehumbert
Förderung
- UKRI | Engineering and Physical Sciences Research Council (EPSRC)
- German Academic Scholarship Foundation
- German Academic Exchange Service (DAAD)
- EC | European Research Council (ERC)
Zitieren
APA
Hofmann, V., Weissweiler, L., Mortensen, D. R., Schütze, H., & Pierrehumbert, J. B. (2025). Derivational morphology reveals analogical generalization in large language models. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2423232122
Vancouver
Hofmann V, Weissweiler L, Mortensen DR, Schütze H, Pierrehumbert JB. Derivational morphology reveals analogical generalization in large language models. Proceedings of the National Academy of Sciences. 2025. doi:10.1073/pnas.2423232122
BibTeX
@article{Hofmann2025,
author = {Hofmann, Valentin and Weissweiler, Leonie and Mortensen, David R and Schütze, Hinrich and Pierrehumbert, Janet B},
title = {Derivational morphology reveals analogical generalization in large language models},
journal = {Proceedings of the National Academy of Sciences},
year = {2025},
doi = {10.1073/pnas.2423232122}
}
EUVIMED