LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-03 · 47 (3) · 1877-1893
DOI: 10.1109/tpami.2024.3507000 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-03. Verfasst von Peng Xu und 9 weiteren Autor:innen. Die Arbeit wurde bisher 89-mal zitiert (Crossref) und nennt 106 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Key R&D Program of China, General Research Fund of Hong Kong.
Abstract (Originalsprache)
Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building an LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of 13 representative LVLMs such as InstructBLIP and LLaVA, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates five categories of multimodal capabilities of LVLMs such as visual question answering and object hallucination on 42 in-domain text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study investigates how specific features of LVLMs such as model configurations, modality alignment mechanisms, and training data affect the multimodal understanding. By conducting a comprehensive comparison of these features on quantitative and arena evaluation, our study uncovers several innovative findings, which establish a fundamental framework for the development and evaluation of innovative strategies aimed at enhancing multimodal techniques. Our LVLM-eHub is available at https://github.com/OpenGVLab/Multi-Modality-Arena.
Förderung
- National Key R&D Program of China
- General Research Fund of Hong Kong
Zitieren
APA
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2025). LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3), 1877-1893. https://doi.org/10.1109/tpami.2024.3507000
Vancouver
Xu P, Shao W, Zhang K, Gao P, Liu S, Lei M, et al. LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(3):1877-1893. doi:10.1109/tpami.2024.3507000
BibTeX
@article{Xu2025,
author = {Xu, Peng and Shao, Wenqi and Zhang, Kaipeng and Gao, Peng and Liu, Shuo and Lei, Meng and Meng, Fanqing and Huang, Siyuan and Qiao, Yu and Luo, Ping},
title = {LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {3},
pages = {1877--1893},
doi = {10.1109/tpami.2024.3507000}
}
EUVIMED