EUVIMEDEUVIMED entdecken
Crossref

Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models

Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Chen Wang, Jiahua Dong, Wangbo Yu, Ge Zhang, Xiang Li, Ian Reid und 1 weitere

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (4) · 2616-2624

DOI: 10.1609/aaai.v40i4.37249 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Meng Cao und 12 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advancements in Large Video Language Models (LVLMs) have highlighted their potential for multi-modal understanding, yet evaluating their factual grounding in videos remains a critical unsolved challenge. To address this gap, we introduce Video SimpleQA, the first comprehensive benchmark tailored for factuality evaluation in video contexts. Our work differs from existing video benchmarks through the following key features: 1) Knowledge required: demanding integration of external knowledge beyond the video’s explicit narrative; 2) Multi-hop fact-seeking question: Each question involves multiple explicit facts and requires strict factual grounding without hypothetical or subjective inferences. We include per-hop single-fact-based sub-QAs alongside final QAs to enable fine-grained, step-by-step evaluation; 3) Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance; 4) Temporal grounded required: Requiring answers to rely on one or more temporal segments in videos, rather than single frames. We extensively evaluate 33 state-of-the-art LVLMs and summarize key findings as follows: 1) Current LVLMs exhibit notable deficiencies in factual adherence, with the best-performing model o3 merely achieving an F-score of 66.3%; 2) Most LVLMs are overconfident in what they generate, with self-stated confidence exceeding actual accuracy; 3) Retrieval-Augmented Generation demonstrates consistent improvements at the cost of additional inference time overhead; 4) Multi-hop QA demonstrates substantially degraded performance compared to single-hop sub-QAs, with first-hop object/event recognition emerging as the primary bottleneck. We position Video SimpleQA as the cornerstone benchmark for video factuality assessment, aiming to steer LVLM development toward verifiable grounding in real-world contexts.

Zitieren

APA

Cao, M., Hu, P., Wang, Y., Gu, J., Tang, H., Zhao, H., Wang, C., Dong, J., Yu, W., Zhang, G., Li, X., Reid, I., & Liang, X. (2026). Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(4), 2616-2624. https://doi.org/10.1609/aaai.v40i4.37249

Vancouver

Cao M, Hu P, Wang Y, Gu J, Tang H, Zhao H, et al. Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(4):2616-2624. doi:10.1609/aaai.v40i4.37249

BibTeX
@article{Cao2026,
  author = {Cao, Meng and Hu, Pengfei and Wang, Yingyao and Gu, Jihao and Tang, Haoran and Zhao, Haoze and Wang, Chen and Dong, Jiahua and Yu, Wangbo and Zhang, Ge and Li, Xiang and Reid, Ian and Liang, Xiaodan},
  title = {Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {4},
  pages = {2616--2624},
  doi = {10.1609/aaai.v40i4.37249}
}
Kostenlos starten →