Valley: Video Assistant with Large Language Model Enhanced Ability
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.07.2026 · 22 (7) · 1-25
DOI: 10.1145/3796716 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 24.06.2026. Verfasst von Ruipu Luo und 7 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large Language Models (LLMs), with remarkable conversational capabilities, have emerged as AI assistants that can handle both visual and textual modalities. However, their effectiveness in joint video-language understanding has not been extensively explored. In the article, we introduce Valley , a multi-modal foundation model designed to enable enhanced video comprehension and instruction-following capabilities. To this end, we construct two datasets, namely “ Valley-702k ” and “ Valley-instruct-73k ,” to cover a diverse range of video–text alignment and video-based instruction tasks, such as multi-shot captions, long video descriptions, action recognition, causal inference, and so on. Then, we adopt ViT-L/14 as the vision encoder and explore three different temporal modeling modules to learn multifaceted features for enhanced video understanding. In addition, we implement a two-phase training approach for Valley: the first phase focuses solely on training the projection module to facilitate the LLM’s capacity to understand visual input, and the second phase jointly trains the projection module and the LLM to improve their instruction following ability. Extensive experiments demonstrate that Valley has the potential to serve as an effective video assistant, simplifying complex video understanding tasks. Our code and data are publicly available at https://github.com/RupertLuo/Valley .
Zitieren
APA
Luo, R., Zhao, Z., Yang, M., Yang, Z., Qiu, M., Wei, Z., Wang, Y., & Chen, C. (2026). Valley: Video Assistant with Large Language Model Enhanced Ability. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(7), 1-25. https://doi.org/10.1145/3796716
Vancouver
Luo R, Zhao Z, Yang M, Yang Z, Qiu M, Wei Z, et al. Valley: Video Assistant with Large Language Model Enhanced Ability. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(7):1-25. doi:10.1145/3796716
BibTeX
@article{Luo2026,
author = {Luo, Ruipu and Zhao, Ziwang and Yang, Min and Yang, Zheming and Qiu, Minghui and Wei, Zhongyu and Wang, Yanhao and Chen, Cen},
title = {Valley: Video Assistant with Large Language Model Enhanced Ability},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {7},
pages = {1--25},
doi = {10.1145/3796716}
}
EUVIMED