Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (4) · 2616-2624
DOI: 10.1609/aaai.v40i4.37249 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Meng Cao und 12 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).
Abstract (Originalsprache)
Recent advancements in Large Video Language Models (LVLMs) have highlighted their potential for multi-modal understanding, yet evaluating their factual grounding in videos remains a critical unsolved challenge. To address this gap, we introduce Video SimpleQA, the first comprehensive benchmark tailored for factuality evaluation in video contexts. Our work differs from existing video benchmarks through the following key features: 1) Knowledge required: demanding integration of external knowledge beyond the video’s explicit narrative; 2) Multi-hop fact-seeking question: Each question involves multiple explicit facts and requires strict factual grounding without hypothetical or subjective inferences. We include per-hop single-fact-based sub-QAs alongside final QAs to enable fine-grained, step-by-step evaluation; 3) Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance; 4) Temporal grounded required: Requiring answers to rely on one or more temporal segments in videos, rather than single frames. We extensively evaluate 33 state-of-the-art LVLMs and summarize key findings as follows: 1) Current LVLMs exhibit notable deficiencies in factual adherence, with the best-performing model o3 merely achieving an F-score of 66.3%; 2) Most LVLMs are overconfident in what they generate, with self-stated confidence exceeding actual accuracy; 3) Retrieval-Augmented Generation demonstrates consistent improvements at the cost of additional inference time overhead; 4) Multi-hop QA demonstrates substantially degraded performance compared to single-hop sub-QAs, with first-hop object/event recognition emerging as the primary bottleneck. We position Video SimpleQA as the cornerstone benchmark for video factuality assessment, aiming to steer LVLM development toward verifiable grounding in real-world contexts.
Zitieren
APA
Cao, M., Hu, P., Wang, Y., Gu, J., Tang, H., Zhao, H., Wang, C., Dong, J., Yu, W., Zhang, G., Li, X., Reid, I., & Liang, X. (2026). Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(4), 2616-2624. https://doi.org/10.1609/aaai.v40i4.37249
Vancouver
Cao M, Hu P, Wang Y, Gu J, Tang H, Zhao H, et al. Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(4):2616-2624. doi:10.1609/aaai.v40i4.37249
BibTeX
@article{Cao2026,
author = {Cao, Meng and Hu, Pengfei and Wang, Yingyao and Gu, Jihao and Tang, Haoran and Zhao, Haoze and Wang, Chen and Dong, Jiahua and Yu, Wangbo and Zhang, Ge and Li, Xiang and Reid, Ian and Liang, Xiaodan},
title = {Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {4},
pages = {2616--2624},
doi = {10.1609/aaai.v40i4.37249}
}
EUVIMED