EUVIMEDEUVIMED entdecken
Crossref

FreeEdit: Mask-Free Reference-Based Image Editing With Multi-Modal Instruction

Runze He, Kai Ma, Linjiang Huang, Shaofei Huang, Jialin Gao, Xiaoming Wei, Jiao Dai, Jizhong Han, Si Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-03 · 48 (3) · 3319-3334

DOI: 10.1109/tpami.2025.3636582 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-03. Verfasst von Runze He und 8 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 75 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, The Fundamental Research Funds for the Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introducing user-specified visual concepts in image editing is highly practical as these concepts convey the user's intent more precisely than text-based descriptions. We propose FreeEdit, a novel approach for achieving such reference-based image editing, which can accurately reproduce the visual concept from the reference image based on user-friendly language instructions. Our approach leverages the multi-modal instruction encoder to encode language instructions to guide the editing process. This implicit way of locating the editing area eliminates the need for manual editing masks. To enhance the reconstruction of reference details, we introduce the Decoupled Residual Refer-Attention (DRRA) module. This module is designed to integrate fine-grained reference features extracted by a detail extractor into the image editing process in a residual way without interfering with the original self-attention. Given that existing datasets are unsuitable for reference-based image editing tasks, particularly due to the difficulty in constructing image triplets that include a reference image, we curate a high-quality dataset, FreeBench, using a newly developed twice-repainting scheme. FreeBench comprises the images before and after editing, detailed editing instructions, as well as a reference image that maintains the identity of the edited object, encompassing tasks such as object addition, replacement, and deletion. By conducting phased training on FreeBench followed by quality tuning, FreeEdit achieves high-quality zero-shot editing through convenient language instructions. We conduct extensive experiments to evaluate the effectiveness of FreeEdit across multiple task types, demonstrating its superiority over existing methods.

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China
  • The Fundamental Research Funds for the Central Universities
  • Pioneer
  • Leading Goose
  • R&D Program of Zhejiang
  • Beijing Natural Science Foundation
  • Ningbo Science and Technology Innovation 2025 Major Project

Zitieren

APA

He, R., Ma, K., Huang, L., Huang, S., Gao, J., Wei, X., Dai, J., Han, J., & Liu, S. (2026). FreeEdit: Mask-Free Reference-Based Image Editing With Multi-Modal Instruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3), 3319-3334. https://doi.org/10.1109/tpami.2025.3636582

Vancouver

He R, Ma K, Huang L, Huang S, Gao J, Wei X, et al. FreeEdit: Mask-Free Reference-Based Image Editing With Multi-Modal Instruction. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(3):3319-3334. doi:10.1109/tpami.2025.3636582

BibTeX
@article{He2026,
  author = {He, Runze and Ma, Kai and Huang, Linjiang and Huang, Shaofei and Gao, Jialin and Wei, Xiaoming and Dai, Jiao and Han, Jizhong and Liu, Si},
  title = {FreeEdit: Mask-Free Reference-Based Image Editing With Multi-Modal Instruction},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {3},
  pages = {3319--3334},
  doi = {10.1109/tpami.2025.3636582}
}
Kostenlos starten →