EUVIMEDEUVIMED entdecken
Crossref

Memory$^3$: Language Modeling with Explicit Memory

Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, Chenyang Xi, Yu Yu und 4 weitere

Journal of Machine Learning · 21.09.2024 · 3 (3) · 300-346

DOI: 10.4208/jml.240708 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Learning, online erschienen am 21.09.2024. Verfasst von Hongkang Yang und 15 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 133 Literaturangaben. Gefördert von: NSFC Major Research Plan - Interpretable and General Purpose Next-generation Artificial Intelligence of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining “abstract knowledge”. As a preliminary proof of concept, we train from scratch a 2.4 B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named ${\rm Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.

Förderung

  • NSFC Major Research Plan - Interpretable and General Purpose Next-generation Artificial Intelligence of China

Zitieren

APA

Yang, H., Lin, Z., Wang, W., Wu, H., Li, Z., Tang, B., Wei, W., Wang, J., Tang, Z., Song, S., Xi, C., Yu, Y., Chen, K., Xiong, F., Tang, L., & E, W. (2024). Memory$^3$: Language Modeling with Explicit Memory. Journal of Machine Learning, 3(3), 300-346. https://doi.org/10.4208/jml.240708

Vancouver

Yang H, Lin Z, Wang W, Wu H, Li Z, Tang B, et al. Memory$^3$: Language Modeling with Explicit Memory. Journal of Machine Learning. 2024;3(3):300-346. doi:10.4208/jml.240708

BibTeX
@article{Yang2024,
  author = {Yang, Hongkang and Lin, Zehao and Wang, Wenjin and Wu, Hao and Li, Zhiyu and Tang, Bo and Wei, Wenqiang and Wang, Jinbo and Tang, Zeyun and Song, Shichao and Xi, Chenyang and Yu, Yu and Chen, Kai and Xiong, Feiyu and Tang, Linpeng and E, Weinan},
  title = {Memory$^3$: Language Modeling with Explicit Memory},
  journal = {Journal of Machine Learning},
  year = {2024},
  volume = {3},
  number = {3},
  pages = {300--346},
  doi = {10.4208/jml.240708}
}
Kostenlos starten →