VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-02 · 47 (2) · 708-724
DOI: 10.1109/tpami.2024.3479776 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-02. Verfasst von Jing Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 125 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Key Research and Development Program of Jiangsu Province.
Abstract (Originalsprache)
In this paper, we propose the Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multimodal understanding and generation. Unlike widely-studied vision-language pretraining models, VALOR jointly models the relationships among vision, audio, and language in an end-to-end manner. It consists of three separate encoders for single modality representations and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain the VALOR model: Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language, and audio into the same common space, simultaneously building vision-language, audio-language, and audiovisual-language alignment. MGC learns to generate text tokens under conditions of vision, audio, or both. To promote vision-audio-language pretraining research, we construct a large-scale, high-quality tri-modality dataset named VALOR-1M, containing 1 million audible videos with human-annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and generalize to various downstream tasks (e.g., retrieval, captioning, and question answering) with different input modalities (e.g., vision-language, audio-language, and audiovisual-language). VALOR achieves new state-of-the-art performance on a series of public cross-modality benchmarks.
Förderung
- National Natural Science Foundation of China
- Key Research and Development Program of Jiangsu Province
Zitieren
APA
Liu, J., Chen, S., He, X., Guo, L., Zhu, X., Wang, W., & Tang, J. (2025). VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(2), 708-724. https://doi.org/10.1109/tpami.2024.3479776
Vancouver
Liu J, Chen S, He X, Guo L, Zhu X, Wang W, et al. VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(2):708-724. doi:10.1109/tpami.2024.3479776
BibTeX
@article{Liu2025,
author = {Liu, Jing and Chen, Sihan and He, Xingjian and Guo, Longteng and Zhu, Xinxin and Wang, Weining and Tang, Jinhui},
title = {VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {2},
pages = {708--724},
doi = {10.1109/tpami.2024.3479776}
}
EUVIMED