An Empirical Comparison of Discrete Video Tokenization Schemes for Video Question Answering and Video Captioning
Artificial Intelligence and Machine Learning Review · 11.04.2025 · 6 (2) · 27-50
DOI: 10.69987/aimlr.2025.60203 ↗Kurz gefasst
Fachartikel in der Zeitschrift Artificial Intelligence and Machine Learning Review, online erschienen am 11.04.2025. Verfasst von Mingzhuo Yu und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Discrete video tokenization has become a central design choice in recent multimodal large language models, shifting visual understanding from continuous patch features toward compact sequences of code indices. While rapid progress has been demonstrated on generative benchmarks, the impact of tokenizer choice on discriminative downstream tasks such as video question answering and video captioning remains under-quantified. This paper presents a controlled empirical comparison of nine discrete video tokenization schemes — VQ-VAE, VQ-GAN, VideoGPT, MAGVIT, MAGVIT-v2, OmniTokenizer, LARP, TiTok, and Cosmos Tokenizer — evaluated under a shared decoder-only language backbone on five open-ended video question answering benchmarks (MSRVTT-QA, MSVD-QA, ActivityNet-QA, NExT-QA, TGIF-QA) and two captioning benchmarks (MSR-VTT, VATEX). Across 63 tokenizer–dataset pairs, modern lookup-free and finite-scalar quantization schemes outperform the classic VQ-VAE baseline by 7.6 to 9.5 accuracy points on short-video question answering and 10.8 to 12.3 CIDEr points on captioning, while holistic query-based tokens deliver the largest gains on causal and temporal reasoning. A factor analysis separates codebook size from spatial-temporal compression, revealing a moderate sweet spot around 65K codebook entries at an 8×8×4 compression ratio, beyond which downstream accuracy plateaus or regresses. The results supply practical guidance for tokenizer selection in video–language pipelines.
Zitieren
APA
Yu, M., & Li, Z. (2025). An Empirical Comparison of Discrete Video Tokenization Schemes for Video Question Answering and Video Captioning. Artificial Intelligence and Machine Learning Review, 6(2), 27-50. https://doi.org/10.69987/aimlr.2025.60203
Vancouver
Yu M, Li Z. An Empirical Comparison of Discrete Video Tokenization Schemes for Video Question Answering and Video Captioning. Artificial Intelligence and Machine Learning Review. 2025;6(2):27-50. doi:10.69987/aimlr.2025.60203
BibTeX
@article{Yu2025,
author = {Yu, Mingzhuo and Li, Zan},
title = {An Empirical Comparison of Discrete Video Tokenization Schemes for Video Question Answering and Video Captioning},
journal = {Artificial Intelligence and Machine Learning Review},
year = {2025},
volume = {6},
number = {2},
pages = {27--50},
doi = {10.69987/aimlr.2025.60203}
}
EUVIMED