EUVIMEDEUVIMED entdecken
Crossref

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-02 · 47 (2) · 708-724

DOI: 10.1109/tpami.2024.3479776 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-02. Verfasst von Jing Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 125 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Key Research and Development Program of Jiangsu Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we propose the Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multimodal understanding and generation. Unlike widely-studied vision-language pretraining models, VALOR jointly models the relationships among vision, audio, and language in an end-to-end manner. It consists of three separate encoders for single modality representations and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain the VALOR model: Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language, and audio into the same common space, simultaneously building vision-language, audio-language, and audiovisual-language alignment. MGC learns to generate text tokens under conditions of vision, audio, or both. To promote vision-audio-language pretraining research, we construct a large-scale, high-quality tri-modality dataset named VALOR-1M, containing 1 million audible videos with human-annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and generalize to various downstream tasks (e.g., retrieval, captioning, and question answering) with different input modalities (e.g., vision-language, audio-language, and audiovisual-language). VALOR achieves new state-of-the-art performance on a series of public cross-modality benchmarks.

Förderung

  • National Natural Science Foundation of China
  • Key Research and Development Program of Jiangsu Province

Zitieren

APA

Liu, J., Chen, S., He, X., Guo, L., Zhu, X., Wang, W., & Tang, J. (2025). VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(2), 708-724. https://doi.org/10.1109/tpami.2024.3479776

Vancouver

Liu J, Chen S, He X, Guo L, Zhu X, Wang W, et al. VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(2):708-724. doi:10.1109/tpami.2024.3479776

BibTeX
@article{Liu2025,
  author = {Liu, Jing and Chen, Sihan and He, Xingjian and Guo, Longteng and Zhu, Xinxin and Wang, Weining and Tang, Jinhui},
  title = {VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {2},
  pages = {708--724},
  doi = {10.1109/tpami.2024.3479776}
}
Kostenlos starten →