EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitOpen Access · CC BY 4.0

A survey on multimodal large language models

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, Enhong Chen

National Science Review · 14.11.2024 · 11 (12)

DOI: 10.1093/nsr/nwae403 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift National Science Review, online erschienen am 12.11.2024. Verfasst von Shukang Yin und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230026, China. Die Arbeit wurde bisher 870-mal zitiert (Crossref) und nennt 154 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Young Scientists Fund, Natural Science Foundation of Sichuan Province.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of the MLLM, such as writing stories based on images and optical character recognition–free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even outperform GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First, we present the basic formulation of the MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages and scenarios. We continue with multimodal hallucination and extended techniques, including multimodal in-context learning, multimodal chain of thought and LLM-aided visual reasoning. To conclude the paper, we discuss existing challenges and point out promising research directions.

Schlagwörter

Stichwörter der Autor:innen

large language modelmultimodal large language modelvision language model

Autor:innen und Institutionen

  • School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230026, China.
  • State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China.
  • School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China.
  • Tencent YouTu Lab, Shanghai 200233, China.

Förderung

  • National Natural Science Foundation of China
  • Young Scientists Fund
  • Natural Science Foundation of Sichuan Province

Zitieren

APA

Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024). A survey on multimodal large language models. National Science Review, 11(12). https://doi.org/10.1093/nsr/nwae403

Vancouver

Yin S, Fu C, Zhao S, Li K, Sun X, Xu T, et al. A survey on multimodal large language models. National Science Review. 2024;11(12). doi:10.1093/nsr/nwae403

BibTeX
@article{Yin2024,
  author = {Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Li, Ke and Sun, Xing and Xu, Tong and Chen, Enhong},
  title = {A survey on multimodal large language models},
  journal = {National Science Review},
  year = {2024},
  volume = {11},
  number = {12},
  doi = {10.1093/nsr/nwae403}
}
Kostenlos starten →