UMIE: Unified Multimodal Information Extraction with Instruction Tuning
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (17) · 19062-19070
DOI: 10.1609/aaai.v38i17.29873 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Lin Sun und 3 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref).
Abstract (Originalsprache)
Multimodal information extraction (MIE) gains significant attention as the popularity of multimedia content increases. However, current MIE methods often resort to using task-specific model structures, which results in limited generalizability across tasks and underutilizes shared knowledge across MIE tasks. To address these issues, we propose UMIE, a unified multimodal information extractor to unify three MIE tasks as a generation problem using instruction tuning, being able to effectively extract both textual and visual mentions. Extensive experiments show that our single UMIE outperforms various state-of-the-art (SoTA) methods across six MIE datasets on three tasks. Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the zero-shot setting, robustness to instruction variants, and interpretability. Our research serves as an initial step towards a unified MIE model and initiates the exploration into both instruction tuning and large language models within the MIE domain. Our code, data, and model are available at https://github.com/ZUCC-AI/UMIE.
Zitieren
APA
Sun, L., Zhang, K., Li, Q., & Lou, R. (2024). UMIE: Unified Multimodal Information Extraction with Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17), 19062-19070. https://doi.org/10.1609/aaai.v38i17.29873
Vancouver
Sun L, Zhang K, Li Q, Lou R. UMIE: Unified Multimodal Information Extraction with Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(17):19062-19070. doi:10.1609/aaai.v38i17.29873
BibTeX
@article{Sun2024,
author = {Sun, Lin and Zhang, Kai and Li, Qingyuan and Lou, Renze},
title = {UMIE: Unified Multimodal Information Extraction with Instruction Tuning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {17},
pages = {19062--19070},
doi = {10.1609/aaai.v38i17.29873}
}
EUVIMED