EUVIMEDEUVIMED entdecken
Crossref

Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation

Yeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao, Daxin Jiang, Ruihang Miao, Cam-Tu Nguyen

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (34) · 28456-28464

DOI: 10.1609/aaai.v40i34.40075 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yeqin Zhang und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text representation plays a critical role in tasks like clustering, retrieval, and other downstream applications. With the emergence of large language models (LLMs), there is increasing interest in harnessing their capabilities for this purpose. However, most of the LLMs are inherently causal and optimized for next-token prediction, making them suboptimal for producing holistic representations. To address this, recent studies introduced pretext tasks to adapt LLMs for text representation. Most of these tasks, however, rely on token-level prediction objectives, such as the masked next-token prediction (MNTP) used in LLM2Vec. In this work, we explore the untapped potential of context compression as a pretext task for unsupervised adaptation of LLMs. During compression pre-training, the model learns to generate compact memory tokens, which substitute the whole context for downstream sequence prediction. Experiments demonstrate that a well-designed compression objective can significantly enhance LLM-based text representations, outperforming models trained with token-level pretext tasks. Further improvements through contrastive learning produce a strong representation model (LLM2Comp) that outperforms contemporary LLM-based text encoders on a wide range of tasks while being more sample-efficient, requiring significantly less training data.

Zitieren

APA

Zhang, Y., Zhao, Y., Hu, C., Jiao, B., Jiang, D., Miao, R., & Nguyen, C. (2026). Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(34), 28456-28464. https://doi.org/10.1609/aaai.v40i34.40075

Vancouver

Zhang Y, Zhao Y, Hu C, Jiao B, Jiang D, Miao R, et al. Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(34):28456-28464. doi:10.1609/aaai.v40i34.40075

BibTeX
@article{Zhang2026,
  author = {Zhang, Yeqin and Zhao, Yizheng and Hu, Chen and Jiao, Binxing and Jiang, Daxin and Miao, Ruihang and Nguyen, Cam-Tu},
  title = {Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {34},
  pages = {28456--28464},
  doi = {10.1609/aaai.v40i34.40075}
}
Kostenlos starten →