Transformer Based Pluralistic Image Completion With Reduced Information Loss
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-10 · 46 (10) · 6652-6668
DOI: 10.1109/tpami.2024.3384406 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-10. Verfasst von Qiankun Liu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
Transformer based methods have achieved great success in image inpainting recently. However, we find that these solutions regard each pixel as a token, thus suffering from an information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration. 2) They quantize 2563 RGB values to a small number (such as 512) of quantized color values. The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer. To mitigate these issues, we propose a new transformer based framework called "PUT". Specifically, to avoid input downsampling while maintaining computation efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by input quantization, an Un-quantized Transformer is applied. It directly takes features from the P-VQVAE encoder as input without any quantization and only regards the quantized tokens as prediction targets. Furthermore, to make the inpainting process more controllable, we introduce semantic and structural conditions as extra guidance. Extensive experiments show that our method greatly outperforms existing transformer based methods on image fidelity and achieves much higher diversity and better fidelity than state-of-the-art pluralistic inpainting methods on complex large-scale datasets (e.g., ImageNet).
Förderung
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Liu, Q., Jiang, Y., Tan, Z., Chen, D., Fu, Y., Chu, Q., Hua, G., & Yu, N. (2024). Transformer Based Pluralistic Image Completion With Reduced Information Loss. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10), 6652-6668. https://doi.org/10.1109/tpami.2024.3384406
Vancouver
Liu Q, Jiang Y, Tan Z, Chen D, Fu Y, Chu Q, et al. Transformer Based Pluralistic Image Completion With Reduced Information Loss. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(10):6652-6668. doi:10.1109/tpami.2024.3384406
BibTeX
@article{Liu2024,
author = {Liu, Qiankun and Jiang, Yuqi and Tan, Zhentao and Chen, Dongdong and Fu, Ying and Chu, Qi and Hua, Gang and Yu, Nenghai},
title = {Transformer Based Pluralistic Image Completion With Reduced Information Loss},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {10},
pages = {6652--6668},
doi = {10.1109/tpami.2024.3384406}
}
EUVIMED