Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout
Computers · 01.11.2026 · 15 (11) · 769
DOI: 10.3390/computers15110769 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computers, online erschienen am 01.11.2026. Verfasst von Dimitris Karydas und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We present a unified experimental decomposition of transformer prediction that separates information availability, native functional dependence, representational accessibility, and downstream predictive readout. For finite-vocabulary order-k Markov sources, the context penalty is exactly HL−H*=I(Xt+1;St∣Ct(L)) and vanishes for L≥k; all 15 sampled dense kernels show a strictly positive Hk−1−H* gap. Scratch transformers approach the corresponding context-limited optimum but retain optimization-dependent residuals. Post-training state-changing interventions steer the native predictive distribution toward the exact counterfactual oracle, and source-supported alternatives preserve 0.72–0.77-nat tracking gains while matched out-of-state controls remain near zero. In frozen Qwen2.5 representations, repeating the analysis over ten independent vocabulary maps per fixed checkpoint preserves the sharp state accessibility transition at L=k; Pythia-410M provides a cross-family replication. Capacity and sample-size controls show that residual predictive readout loss can arise even when the true state is supplied directly. Finally, a controlled hidden Markov model extension replaces the visible suffix state by a latent Bayesian belief state: exact filtering, scratch prediction, posterior probing, and predictive readout remain separately measurable, while the predictive gap to a 256-observation Bayesian reference decays smoothly rather than at a finite boundary. The results support a bounded four-level account of context sufficiency without equating decodability with native causal use.
Zitieren
APA
Karydas, D., Margaritis, D., & Leligou, H. C. (2026). Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout. Computers, 15(11), 769. https://doi.org/10.3390/computers15110769
Vancouver
Karydas D, Margaritis D, Leligou HC. Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout. Computers. 2026;15(11):769. doi:10.3390/computers15110769
BibTeX
@article{Karydas2026,
author = {Karydas, Dimitris and Margaritis, Dimosthenis and Leligou, Helen C.},
title = {Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout},
journal = {Computers},
year = {2026},
volume = {15},
number = {11},
pages = {769},
doi = {10.3390/computers15110769}
}
EUVIMED