Boundary Screening Bounds the Evidence in Span Masking but Does Not Predict Linear-Probe Representation Quality at Small Scale
Machine Learning and Knowledge Extraction · 09.11.2026 · 8 (11) · 373
DOI: 10.3390/make8110373 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machine Learning and Knowledge Extraction, online erschienen am 09.11.2026. Verfasst von Suwichai Phunsa und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Mahasarakham University.
Abstract (Originalsprache)
Span masking is usually justified informally: contiguous masks are said to force phrase-level rather than token-level inference. We quantify that argument and then test it. A boundary-screening theorem shows that the evidence a context carries about a masked set passes entirely through the block boundaries, so it is bounded by a quantity scaling with the number of blocks rather than of masked tokens. The downstream prediction this motivates, pre-registered with fixed decision rules, is refuted under the tested design. Across a sixteen-fold range of span length at fixed masking ratio, probe accuracy varies by 0.80 points, against 2.0 to 4.3 point changes the same design resolves from auxiliary objectives and training length. Mask geometry therefore bounds the available evidence without predicting linear-probe representation quality at this scale. Two further predictions hold and a third in part: the Gaussian closed form matches direct computation to 10−13 nats, a radial conservation law for cosine objectives holds in a trained transformer to within 3 percent, and an auxiliary masked-language term degrades semantic probes while barely moving lexical ones. The theorem is exact for finite-order Markov chains; its natural-language counterpart is a character-level diagnostic, and the encoders are 4.5 M parameters trained on 74 M tokens, at which a random-initialised baseline is already strong.
Förderung
- Mahasarakham University
Zitieren
APA
Phunsa, S., & Chomsiri, T. (2026). Boundary Screening Bounds the Evidence in Span Masking but Does Not Predict Linear-Probe Representation Quality at Small Scale. Machine Learning and Knowledge Extraction, 8(11), 373. https://doi.org/10.3390/make8110373
Vancouver
Phunsa S, Chomsiri T. Boundary Screening Bounds the Evidence in Span Masking but Does Not Predict Linear-Probe Representation Quality at Small Scale. Machine Learning and Knowledge Extraction. 2026;8(11):373. doi:10.3390/make8110373
BibTeX
@article{Phunsa2026,
author = {Phunsa, Suwichai and Chomsiri, Thawatchai},
title = {Boundary Screening Bounds the Evidence in Span Masking but Does Not Predict Linear-Probe Representation Quality at Small Scale},
journal = {Machine Learning and Knowledge Extraction},
year = {2026},
volume = {8},
number = {11},
pages = {373},
doi = {10.3390/make8110373}
}
EUVIMED