EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

EIM: An effective solution for improving multi-modal large language models

Yuting Bai iD, Tonghua Su, Zixing Bai iD

PLOS One · 2025

DOI: 10.1371/journal.pone.0329590 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Yuting Bai und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Software, Harbin Institute of Technology, Harbin, China. Hauptthemen laut MEDLINE: Language, Models, Theoretical. Die Arbeit nennt 96 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Enabling large language models (LLMs) to have multi-modal capabilities, such as vision-language learning, has become a current research hotspot and the next milestone in LLM development with the advent of models like GPT4. The basic structure of current multi-modal LLMs usually includes three parts: the image encoder for extracting visual features, the s emantic space t ransformation network ST for aligning the multi-modal semantic spaces, and LLM for generating text. Current works on multi-modal LLMs primarily focus on enhancing performance by utilizing larger image encoders and LLMs, and designing more complex fine-tuning methods and STs, which results in an escalation of model parameters. In this paper, we propose EIM , a novel e ffective solution for i mproving the performance of m ulti-modal large language models from the perspective of training process which reduces the need to introduce new parameters and modify the model structure, and is ignored and less explored in current research. EIM includes corresponding improvement measures in the image encoder, ST, and LLM. To validate EIM, we first apply it to ClipCap and conduct experiments on the COCO Caption dataset. Secondly, we extend EIM to the multi-modal LLMs, such as LLaMA-Adapter and LaVIN, and evaluate them on the ScienceQA dataset. Finally, we also conduct multi-modal chatbot experiments with the EIM enhanced LaVIN and evaluate it on the MME benchmark. The COCO Caption dataset experimental results of ClipCap eim , which is a model that applies EIM on the ClipCap small , show the 1.75% performance improvement when compared to those of ClipCap large , which has 3.13 times the number of parameters of ClipCap eim . The experimental results on the ScienceQA dataset and MME benchmark show that EIM can achieve competitive performance with 7B model parameters when compared to the 13B multi-modal LLMs, which confirms the effective performance improvement of EIM for multi-modal LLMs.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageModels, TheoreticalHumansSemanticsAlgorithmsLarge Language Models

Autor:innen und Institutionen

  • School of Software, Harbin Institute of Technology, Harbin, China.
  • School of Computer Science and Technology, Fudan University, Shanghai, China.

ORCID-Profile: Yuting Bai, Zixing Bai

Zitieren

APA

Bai, Y., Su, T., & Bai, Z. (2025). EIM: An effective solution for improving multi-modal large language models. PLOS One. https://doi.org/10.1371/journal.pone.0329590

Vancouver

Bai Y, Su T, Bai Z. EIM: An effective solution for improving multi-modal large language models. PLOS One. 2025. doi:10.1371/journal.pone.0329590

BibTeX
@article{Bai2025,
  author = {Bai, Yuting and Su, Tonghua and Bai, Zixing},
  title = {EIM: An effective solution for improving multi-modal large language models},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0329590}
}
Kostenlos starten →