M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (7) · 7233-7241
DOI: 10.1609/aaai.v38i7.28552 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Ning Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref).
Abstract (Originalsprache)
Document layout analysis is a crucial step for intelligent document understanding. However, many existing methods primarily focus on the visual aspects and overlook the textual features of documents. Although document pre-trained models utilize multi-modal features during the pre-training phase, they tend to operate as a unimodal pipeline when it comes to layout analysis tasks. Furthermore, current multi-modal methods perform worse than unimodal detectors on complex layout analysis datasets. To address these limitations, we propose an effective and pluggable multi-modal fusion approach named M2Doc, which fuses visual and textual features for better layout detection. M2Doc contains two pluggable multi-modal fusion modules, early-fusion and late-fusion, which align and fuse visual and textual features at the pixel level and block level. Benefitting from the concision and effectiveness of M2Doc, it can be easily applied to various detectors for better layout detection, including two-stage and end-to-end object detectors. Our experimental results demonstrate significant performance improvements in detectors equipped with M2Doc on datasets such as DocLayNet (+11.3 mAP) and M6Doc (+1.9 mAP). Furthermore, through the integration of the DINO detector with M2Doc, we achieve state-of-the-art results on DocLayNet (89.0 mAP), M6Doc (69.9 mAP), and PubLayNet (95.5 mAP). The code will be publicly released at https://github.com/johnning2333/M2Doc.
Zitieren
APA
Zhang, N., Cheng, H., Chen, J., Jiang, Z., Huang, J., Xue, Y., & Jin, L. (2024). M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7), 7233-7241. https://doi.org/10.1609/aaai.v38i7.28552
Vancouver
Zhang N, Cheng H, Chen J, Jiang Z, Huang J, Xue Y, et al. M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(7):7233-7241. doi:10.1609/aaai.v38i7.28552
BibTeX
@article{Zhang2024,
author = {Zhang, Ning and Cheng, Hiuyi and Chen, Jiayu and Jiang, Zongyuan and Huang, Jun and Xue, Yang and Jin, Lianwen},
title = {M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {7},
pages = {7233--7241},
doi = {10.1609/aaai.v38i7.28552}
}
EUVIMED