Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction
Applied Sciences · 18.05.2023 · 13 (10) · 6178
DOI: 10.3390/app13106178 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 18.05.2023. Verfasst von Min Zuo und 5 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Technology R&D Program of China, Natural Science Foundation of China, Beijing Natural Science Foundation.
Abstract (Originalsprache)
Multimodal Named Entity Recognition (MNER) and multimodal Relationship Extraction (MRE) play an important role in processing multimodal data and understanding entity relationships across textual and visual domains. However, irrelevant image information may introduce noise that misleads the recognition of information. Additionally, visual and semantic features originate from different modalities, and modal disparity hinders semantic alignment. Therefore, this paper proposes the Visual Description Augmentation Integration Network (VDAIN), which introduces an image description generation technique that allows semantic features generated from image descriptions to be presented in the same modality as the semantic features of textual information. This not only reduces the modal gap but also captures more accurately the high-level semantic information and underlying visual structure in the images. To filter out the modal noise, we use VDAIN to adaptively fuse visual features, semantic features of image descriptions, and textual information, thus eliminating irrelevant modal noise. The F1 score of the proposed model in this paper reaches 75.8% and 87.78% for the MNER task and 82.54% for the MRE task on the three public data sets, respectively, which are significantly better than the baseline model. The experimental results demonstrate the effectiveness of the proposed method in solving the modal noise and modal gap problems.
Förderung
- National Key Technology R&D Program of China
- Natural Science Foundation of China
- Beijing Natural Science Foundation
- Project of Beijing Municipal University Teacher Team Construction Support Plan
- Humanity and Social Science Youth Foundation of Ministry of Education of China
Zitieren
APA
Zuo, M., Wang, Y., Dong, W., Zhang, Q., Cai, Y., & Kong, J. (2023). Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction. Applied Sciences, 13(10), 6178. https://doi.org/10.3390/app13106178
Vancouver
Zuo M, Wang Y, Dong W, Zhang Q, Cai Y, Kong J. Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction. Applied Sciences. 2023;13(10):6178. doi:10.3390/app13106178
BibTeX
@article{Zuo2023,
author = {Zuo, Min and Wang, Yingjun and Dong, Wei and Zhang, Qingchuan and Cai, Yuanyuan and Kong, Jianlei},
title = {Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction},
journal = {Applied Sciences},
year = {2023},
volume = {13},
number = {10},
pages = {6178},
doi = {10.3390/app13106178}
}
EUVIMED