When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression
IEEE Signal Processing Letters · 2026 · 33 · 1716-1720
DOI: 10.1109/lsp.2026.3673193 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Signal Processing Letters, 2026. Verfasst von Pingping Zhang und 8 weiteren Autor:innen. Die Arbeit nennt 14 Literaturangaben. Gefördert von: Pioneer and Leading +X” Science and Technology Plan of Zhejiang Province, RGC General Research Fund, NSFC/RGC Joint Research Scheme.
Abstract (Originalsprache)
Traditional video compression methods perform well at high bitrates but struggle to preserve fine-grained semantic information at low bitrates. Recently, with the blossoming of Multimodal Large Language Models (MLLMs), Cross-modal compression techniques offer prospective solutions for improving video compression under low-bitrate conditions. In this paper, we propose a unified Cross-Modality Video Compression (CMVC) framework that integrates multimodal representations and video generative models. The encoder disentangles video into spatial and temporal components, which are mapped to compact cross-modal representations using MLLMs. During decoding, different encoding-decoding modes are employed to acquire various video reconstruction qualities, including Text-Text-to-Video (TT2V) for semantic preservation and Image-Text-to-Video (IT2V) for perceptual consistency. Additionally, we elaborate on an efficient frame interpolation model using Low-Rank Adaptation (LoRA) to improve the perceptual quality. Experimental results demonstrate that TT2V achieves effective semantic reconstruction, while IT2V ensures competitive perceptual consistency. These findings suggest the potential of leveraging multimodal priors to improve video compression, offering promising future research directions.
Förderung
- Pioneer and Leading +X” Science and Technology Plan of Zhejiang Province
- RGC General Research Fund
- NSFC/RGC Joint Research Scheme
- ITF Project
- EU Horizon Project ELIAS
- FIS Project GUIDANCE
Zitieren
APA
Zhang, P., Li, J., Chen, K., Wang, M., Xu, L., Li, H., Sebe, N., Kwong, S., & Wang, S. (2026). When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression. IEEE Signal Processing Letters, 33, 1716-1720. https://doi.org/10.1109/lsp.2026.3673193
Vancouver
Zhang P, Li J, Chen K, Wang M, Xu L, Li H, et al. When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression. IEEE Signal Processing Letters. 2026;33:1716-1720. doi:10.1109/lsp.2026.3673193
BibTeX
@article{Zhang2026,
author = {Zhang, Pingping and Li, Jinlong and Chen, Kecheng and Wang, Meng and Xu, Long and Li, Haoliang and Sebe, Nicu and Kwong, Sam and Wang, Shiqi},
title = {When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression},
journal = {IEEE Signal Processing Letters},
year = {2026},
volume = {33},
pages = {1716--1720},
doi = {10.1109/lsp.2026.3673193}
}
EUVIMED