VRT: A Video Restoration Transformer
IEEE Transactions on Image Processing · 2024 · 33 · 2171-2182
DOI: 10.1109/tip.2024.3372454 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2024. Verfasst von Jingyun Liang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 258-mal zitiert (Crossref) und nennt 77 Literaturangaben. Gefördert von: Eidgenössische Technische Hochschule Zürich (ETH Zurich) Fund, Huawei Technologies Oy (Finland) Project, China Scholarship Council.
Abstract (Originalsprache)
Video restoration aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which are restricted by frame-by-frame restoration. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction ability. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal reciprocal self attention (TRSA) and parallel warping. TRSA divides the video into small clips, on which reciprocal attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins (up to 2.16dB) on fourteen benchmark datasets. The codes are available at https://github.com/JingyunLiang/VRT.
Förderung
- Eidgenössische Technische Hochschule Zürich (ETH Zurich) Fund
- Huawei Technologies Oy (Finland) Project
- China Scholarship Council
- Amazon Web Services
- Alexander von Humboldt Foundation
Zitieren
APA
Liang, J., Cao, J., Fan, Y., Zhang, K., Ranjan, R., Li, Y., Timofte, R., & Van Gool, L. (2024). VRT: A Video Restoration Transformer. IEEE Transactions on Image Processing, 33, 2171-2182. https://doi.org/10.1109/tip.2024.3372454
Vancouver
Liang J, Cao J, Fan Y, Zhang K, Ranjan R, Li Y, et al. VRT: A Video Restoration Transformer. IEEE Transactions on Image Processing. 2024;33:2171-2182. doi:10.1109/tip.2024.3372454
BibTeX
@article{Liang2024,
author = {Liang, Jingyun and Cao, Jiezhang and Fan, Yuchen and Zhang, Kai and Ranjan, Rakesh and Li, Yawei and Timofte, Radu and Van Gool, Luc},
title = {VRT: A Video Restoration Transformer},
journal = {IEEE Transactions on Image Processing},
year = {2024},
volume = {33},
pages = {2171--2182},
doi = {10.1109/tip.2024.3372454}
}
EUVIMED