Tokenization Changes Meaning in Large Language Models: Evidence from Chinese
Computational Linguistics · 01.09.2025 · 51 (3) · 785-814
DOI: 10.1162/coli_a_00557 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computational Linguistics, online erschienen am 01.09.2025. Verfasst von David A. Haslett. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 76 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Large language models segment many words into multiple tokens, and there is mixed evidence as to whether tokenization affects how state-of-the-art models represent meanings. Chinese characters present an opportunity to investigate this issue: They contain semantic radicals, which often convey useful information; characters with the same semantic radical tend to begin with the same one or two bytes (when using UTF-8 encodings); and tokens are common strings of bytes, so characters with the same radical often begin with the same token. This study asked GPT-4, GPT-4o, and Llama 3 whether characters contain the same semantic radical, elicited semantic similarity ratings, and conducted odd-one-out tasks (i.e., which character is not like the others). In all cases, misalignment between tokens and radicals systematically corrupted representations of Chinese characters. In experiments comparing characters represented by single tokens to multi-token characters, the models were less accurate for single-token characters, which suggests that segmenting words into fewer, longer tokens obscures valuable information in word form and will not resolve the problems introduced by tokenization. In experiments with 12 European languages, misalignment between tokens and suffixes systematically corrupted categorization of words by all three models, which suggests that the tendency to treat malformed tokens like linguistic units is pervasive.
Zitieren
APA
Haslett, D. A. (2025). Tokenization Changes Meaning in Large Language Models: Evidence from Chinese. Computational Linguistics, 51(3), 785-814. https://doi.org/10.1162/coli_a_00557
Vancouver
Haslett DA. Tokenization Changes Meaning in Large Language Models: Evidence from Chinese. Computational Linguistics. 2025;51(3):785-814. doi:10.1162/coli_a_00557
BibTeX
@article{Haslett2025,
author = {Haslett, David A.},
title = {Tokenization Changes Meaning in Large Language Models: Evidence from Chinese},
journal = {Computational Linguistics},
year = {2025},
volume = {51},
number = {3},
pages = {785--814},
doi = {10.1162/coli_a_00557}
}
EUVIMED