EUVIMEDEUVIMED entdecken
Crossref

When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression

Pingping Zhang, Jinlong Li, Kecheng Chen, Meng Wang, Long Xu, Haoliang Li, Nicu Sebe, Sam Kwong, Shiqi Wang

IEEE Signal Processing Letters · 2026 · 33 · 1716-1720

DOI: 10.1109/lsp.2026.3673193 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Signal Processing Letters, 2026. Verfasst von Pingping Zhang und 8 weiteren Autor:innen. Die Arbeit nennt 14 Literaturangaben. Gefördert von: Pioneer and Leading +X” Science and Technology Plan of Zhejiang Province, RGC General Research Fund, NSFC/RGC Joint Research Scheme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Traditional video compression methods perform well at high bitrates but struggle to preserve fine-grained semantic information at low bitrates. Recently, with the blossoming of Multimodal Large Language Models (MLLMs), Cross-modal compression techniques offer prospective solutions for improving video compression under low-bitrate conditions. In this paper, we propose a unified Cross-Modality Video Compression (CMVC) framework that integrates multimodal representations and video generative models. The encoder disentangles video into spatial and temporal components, which are mapped to compact cross-modal representations using MLLMs. During decoding, different encoding-decoding modes are employed to acquire various video reconstruction qualities, including Text-Text-to-Video (TT2V) for semantic preservation and Image-Text-to-Video (IT2V) for perceptual consistency. Additionally, we elaborate on an efficient frame interpolation model using Low-Rank Adaptation (LoRA) to improve the perceptual quality. Experimental results demonstrate that TT2V achieves effective semantic reconstruction, while IT2V ensures competitive perceptual consistency. These findings suggest the potential of leveraging multimodal priors to improve video compression, offering promising future research directions.

Förderung

  • Pioneer and Leading +X” Science and Technology Plan of Zhejiang Province
  • RGC General Research Fund
  • NSFC/RGC Joint Research Scheme
  • ITF Project
  • EU Horizon Project ELIAS
  • FIS Project GUIDANCE

Zitieren

APA

Zhang, P., Li, J., Chen, K., Wang, M., Xu, L., Li, H., Sebe, N., Kwong, S., & Wang, S. (2026). When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression. IEEE Signal Processing Letters, 33, 1716-1720. https://doi.org/10.1109/lsp.2026.3673193

Vancouver

Zhang P, Li J, Chen K, Wang M, Xu L, Li H, et al. When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression. IEEE Signal Processing Letters. 2026;33:1716-1720. doi:10.1109/lsp.2026.3673193

BibTeX
@article{Zhang2026,
  author = {Zhang, Pingping and Li, Jinlong and Chen, Kecheng and Wang, Meng and Xu, Long and Li, Haoliang and Sebe, Nicu and Kwong, Sam and Wang, Shiqi},
  title = {When Video Compression Meets Multimodal Large Language Models: A Unified Paradigm for Cross-Modality Video Compression},
  journal = {IEEE Signal Processing Letters},
  year = {2026},
  volume = {33},
  pages = {1716--1720},
  doi = {10.1109/lsp.2026.3673193}
}
Kostenlos starten →