EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

When Calibration Data Become Training Data: Specialization and Self-Generated Text in Learned Low-Rank Compression of Large Language Models

Van Tien Pham

Informatics · 19.11.2026 · 13 (11) · 201

DOI: 10.3390/informatics13110201 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Informatics, online erschienen am 19.11.2026. Verfasst von Van Tien Pham. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Low-rank compression shrinks large language models by factorizing their weights, guided by a small calibration set. When the factorization is learned by distillation, as in learned subspace projections (LSP), the calibration set becomes training data, yet is still chosen by convention. We compressed OPT-1.3B, Llama-3.2-1B, and Qwen3-1.7B by 30% and 50% with LSP, calibrated on six sources at a matched budget (WikiText-2, C4, SlimPajama, Alpaca, and two kinds of self-generated text) over three seeds, and repeated the comparison on Llama-3.1-8B and with a second compressor. Training removed most of the up to 600-fold perplexity differences between untrained models but turned them into specialization: each source best preserved its own text, and WikiText-2, the customary choice, gave the least general models. Zero-shot accuracy slightly favored instruction data, through content rather than template. Self-generated documents gave the most uniformly faithful Llama models without external data, whereas exact samples of the model, whose expected training objective is the sequence-level divergence, failed on Qwen3-1.7B. The initialization’s domain survived training, and compressed Llama models collapsed without their calibration beginning-of-sequence token. With every source, compression lost most instruction following and multi-step reasoning. These results indicate that calibration sources should be reported with every perplexity.

Zitieren

APA

Pham, V. T. (2026). When Calibration Data Become Training Data: Specialization and Self-Generated Text in Learned Low-Rank Compression of Large Language Models. Informatics, 13(11), 201. https://doi.org/10.3390/informatics13110201

Vancouver

Pham VT. When Calibration Data Become Training Data: Specialization and Self-Generated Text in Learned Low-Rank Compression of Large Language Models. Informatics. 2026;13(11):201. doi:10.3390/informatics13110201

BibTeX
@article{Pham2026,
  author = {Pham, Van Tien},
  title = {When Calibration Data Become Training Data: Specialization and Self-Generated Text in Learned Low-Rank Compression of Large Language Models},
  journal = {Informatics},
  year = {2026},
  volume = {13},
  number = {11},
  pages = {201},
  doi = {10.3390/informatics13110201}
}
Kostenlos starten →