EUVIMEDEUVIMED entdecken
Crossref

An MLLM-Assisted Denoising Multiscale Visual Fusion Framework for MNER-MI

Xing Li, Huilong Yu, Xiongnan Jin, Ding Zhang

IEEE Transactions on Computational Social Systems · 2026-08 · 13 (4) · 4886-4902

DOI: 10.1109/tcss.2026.3689752 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Computational Social Systems, 2026-08. Verfasst von Xing Li und 3 weiteren Autor:innen. Die Arbeit nennt 39 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, State Key Laboratory of Radio Frequency Heterogeneous Integration.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal named entity recognition (MNER) has become a prominent research area for information extraction from social media content. Recent statistics indicate a growing proportion of social media posts containing multiple images. However, most existing methods focus primarily on single-image scenarios. To bridge this gap, we propose a multimodal large language model-assisted denoising multiscale visual fusion framework (LD-MVF) for multiimage MNER (MNER-MI). Our framework follows a coordinated pipeline of filtering, enhancement, and fusion to systematically address the unique challenges of multiimage scenarios. Specifically, to reduce noise from irrelevant images, we introduce an image–text similarity discriminator trained with human feedback, which provides dynamic weights for visual feature aggregation. To enrich the often sparse textual context, we leverage a multimodal large language model (MLLM) to directly generate grounded auxiliary knowledge from the most relevant images, avoiding information loss caused by textual descriptions. Furthermore, we design an adaptive hierarchical fusion module that extracts multiscale visual features from multiple images and injects them as visual prefixes to enhance textual representations through a dynamic gating mechanism. These three components work cooperatively: the discriminator filters out noise to provide cleaner visual input, the MLLM enhances textual semantics with visually-grounded knowledge, and the fusion module integrates multiscale features for fine-grained cross-modal alignment. Extensive experiments demonstrate that LD-MVF achieves state-of-the-art performance on both the MNER-MI and MNER-MI-Plus datasets.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • State Key Laboratory of Radio Frequency Heterogeneous Integration
  • Scientific Foundation for Youth Scholars of Shenzhen University

Zitieren

APA

Li, X., Yu, H., Jin, X., & Zhang, D. (2026). An MLLM-Assisted Denoising Multiscale Visual Fusion Framework for MNER-MI. IEEE Transactions on Computational Social Systems, 13(4), 4886-4902. https://doi.org/10.1109/tcss.2026.3689752

Vancouver

Li X, Yu H, Jin X, Zhang D. An MLLM-Assisted Denoising Multiscale Visual Fusion Framework for MNER-MI. IEEE Transactions on Computational Social Systems. 2026;13(4):4886-4902. doi:10.1109/tcss.2026.3689752

BibTeX
@article{Li2026,
  author = {Li, Xing and Yu, Huilong and Jin, Xiongnan and Zhang, Ding},
  title = {An MLLM-Assisted Denoising Multiscale Visual Fusion Framework for MNER-MI},
  journal = {IEEE Transactions on Computational Social Systems},
  year = {2026},
  volume = {13},
  number = {4},
  pages = {4886--4902},
  doi = {10.1109/tcss.2026.3689752}
}
Kostenlos starten →