Audio-guided implicit neural representation for local image stylization
Computational Visual Media · 2024-12 · 10 (6) · 1185-1204
DOI: 10.1007/s41095-024-0413-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computational Visual Media, 2024-12. Verfasst von Seung Hyun Lee und 9 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We present a novel framework for audio-guided localized image stylization. Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene. We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space. We then utilize an implicit neural representation (INR) along with the predicted localization map to stylize the target based on sound information. The INR manipulates local pixel values to be semantically consistent with the provided audio input. Our experiments show that the proposed framework outperforms other audio-guided stylization methods. Moreover, we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.
Zitieren
APA
Lee, S. H., Kim, S., Byeon, W., Oh, G., In, S., Park, H., Yoon, S. H., Hong, S., Kim, J., & Kim, S. (2024). Audio-guided implicit neural representation for local image stylization. Computational Visual Media, 10(6), 1185-1204. https://doi.org/10.1007/s41095-024-0413-5
Vancouver
Lee SH, Kim S, Byeon W, Oh G, In S, Park H, et al. Audio-guided implicit neural representation for local image stylization. Computational Visual Media. 2024;10(6):1185-1204. doi:10.1007/s41095-024-0413-5
BibTeX
@article{Lee2024,
author = {Lee, Seung Hyun and Kim, Sieun and Byeon, Wonmin and Oh, Gyeongrok and In, Sumin and Park, Hyeongcheol and Yoon, Sang Ho and Hong, Sung-Hee and Kim, Jinkyu and Kim, Sangpil},
title = {Audio-guided implicit neural representation for local image stylization},
journal = {Computational Visual Media},
year = {2024},
volume = {10},
number = {6},
pages = {1185--1204},
doi = {10.1007/s41095-024-0413-5}
}
EUVIMED