EUVIMEDEUVIMED entdecken
Crossref

UMIE: Unified Multimodal Information Extraction with Instruction Tuning

Lin Sun, Kai Zhang, Qingyuan Li, Renze Lou

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (17) · 19062-19070

DOI: 10.1609/aaai.v38i17.29873 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Lin Sun und 3 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal information extraction (MIE) gains significant attention as the popularity of multimedia content increases. However, current MIE methods often resort to using task-specific model structures, which results in limited generalizability across tasks and underutilizes shared knowledge across MIE tasks. To address these issues, we propose UMIE, a unified multimodal information extractor to unify three MIE tasks as a generation problem using instruction tuning, being able to effectively extract both textual and visual mentions. Extensive experiments show that our single UMIE outperforms various state-of-the-art (SoTA) methods across six MIE datasets on three tasks. Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the zero-shot setting, robustness to instruction variants, and interpretability. Our research serves as an initial step towards a unified MIE model and initiates the exploration into both instruction tuning and large language models within the MIE domain. Our code, data, and model are available at https://github.com/ZUCC-AI/UMIE.

Zitieren

APA

Sun, L., Zhang, K., Li, Q., & Lou, R. (2024). UMIE: Unified Multimodal Information Extraction with Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17), 19062-19070. https://doi.org/10.1609/aaai.v38i17.29873

Vancouver

Sun L, Zhang K, Li Q, Lou R. UMIE: Unified Multimodal Information Extraction with Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(17):19062-19070. doi:10.1609/aaai.v38i17.29873

BibTeX
@article{Sun2024,
  author = {Sun, Lin and Zhang, Kai and Li, Qingyuan and Lou, Renze},
  title = {UMIE: Unified Multimodal Information Extraction with Instruction Tuning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {17},
  pages = {19062--19070},
  doi = {10.1609/aaai.v38i17.29873}
}
Kostenlos starten →