Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-04 · 48 (4) · 3970-3987
DOI: 10.1109/tpami.2025.3642842 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-04. Verfasst von Jiayang Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 75 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Image fusion aims to blend complementary information from diverse sensing modalities, yet most current methods lack robustness in complex fusion scenarios and cannot flexibly accommodate user intent. We present DiTFuse, the first Diffusion-Transformer (DiT) framework for instruction-driven, dynamic fusion control. Guided by natural-language instructions, DiTFuse flexibly blends multimodal content to enable hierarchical and fine-grained control over fusion dynamics. The training phase employs a multi-degrade-mask-image-modeling (M3) strategy, so the network jointly learns cross-modal alignment, modality-invariant restoration, and task-aware feature selection without relying on ideal reference images. A curated, multi-granularity instruction dataset further equips the model with interactive fusion capabilities. DiTFuse unifies infrared-visible, multi-focus, and multi-exposure fusion-as well as text-controlled refinement and downstream tasks-within a single architecture. Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention. The model also supports multi-level user control and zero-shot generalization to other multiimage fusion scenarios, including instruction-conditioned segmentation.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Li, J., Jiang, C., Jiang, J., Liang, P., Ma, J., & Nie, L. (2026). Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(4), 3970-3987. https://doi.org/10.1109/tpami.2025.3642842
Vancouver
Li J, Jiang C, Jiang J, Liang P, Ma J, Nie L. Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(4):3970-3987. doi:10.1109/tpami.2025.3642842
BibTeX
@article{Li2026,
author = {Li, Jiayang and Jiang, Chengjie and Jiang, Junjun and Liang, Pengwei and Ma, Jiayi and Nie, Liqiang},
title = {Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {4},
pages = {3970--3987},
doi = {10.1109/tpami.2025.3642842}
}
EUVIMED