EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout

Dimitris Karydas, Dimosthenis Margaritis, Helen C. Leligou

Computers · 01.11.2026 · 15 (11) · 769

DOI: 10.3390/computers15110769 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 01.11.2026. Verfasst von Dimitris Karydas und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a unified experimental decomposition of transformer prediction that separates information availability, native functional dependence, representational accessibility, and downstream predictive readout. For finite-vocabulary order-k Markov sources, the context penalty is exactly HL−H*=I(Xt+1;St∣Ct(L)) and vanishes for L≥k; all 15 sampled dense kernels show a strictly positive Hk−1−H* gap. Scratch transformers approach the corresponding context-limited optimum but retain optimization-dependent residuals. Post-training state-changing interventions steer the native predictive distribution toward the exact counterfactual oracle, and source-supported alternatives preserve 0.72–0.77-nat tracking gains while matched out-of-state controls remain near zero. In frozen Qwen2.5 representations, repeating the analysis over ten independent vocabulary maps per fixed checkpoint preserves the sharp state accessibility transition at L=k; Pythia-410M provides a cross-family replication. Capacity and sample-size controls show that residual predictive readout loss can arise even when the true state is supplied directly. Finally, a controlled hidden Markov model extension replaces the visible suffix state by a latent Bayesian belief state: exact filtering, scratch prediction, posterior probing, and predictive readout remain separately measurable, while the predictive gap to a 256-observation Bayesian reference decays smoothly rather than at a finite boundary. The results support a bounded four-level account of context sufficiency without equating decodability with native causal use.

Zitieren

APA

Karydas, D., Margaritis, D., & Leligou, H. C. (2026). Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout. Computers, 15(11), 769. https://doi.org/10.3390/computers15110769

Vancouver

Karydas D, Margaritis D, Leligou HC. Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout. Computers. 2026;15(11):769. doi:10.3390/computers15110769

BibTeX
@article{Karydas2026,
  author = {Karydas, Dimitris and Margaritis, Dimosthenis and Leligou, Helen C.},
  title = {Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {769},
  doi = {10.3390/computers15110769}
}
Kostenlos starten →