EUVIMEDEUVIMED entdecken
Crossref

DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding

Hao Feng, Qi Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, Can Huang

Science China Information Sciences · 2024-12 · 67 (12)

DOI: 10.1007/s11432-024-4250-y ↗

Kurz gefasst

Fachartikel in der Zeitschrift Science China Information Sciences, online erschienen am 13.12.2024. Verfasst von Hao Feng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 66 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this work, we present DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2560 × 2560 resolution. Unlike existing studies that either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both the perception and comprehension abilities of our DocPedia, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments are conducted on various publicly available benchmarks and the results confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.

Zitieren

APA

Feng, H., Liu, Q., Liu, H., Tang, J., Zhou, W., Li, H., & Huang, C. (2024). DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Science China Information Sciences, 67(12). https://doi.org/10.1007/s11432-024-4250-y

Vancouver

Feng H, Liu Q, Liu H, Tang J, Zhou W, Li H, et al. DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Science China Information Sciences. 2024;67(12). doi:10.1007/s11432-024-4250-y

BibTeX
@article{Feng2024,
  author = {Feng, Hao and Liu, Qi and Liu, Hao and Tang, Jingqun and Zhou, Wengang and Li, Houqiang and Huang, Can},
  title = {DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding},
  journal = {Science China Information Sciences},
  year = {2024},
  volume = {67},
  number = {12},
  doi = {10.1007/s11432-024-4250-y}
}
Kostenlos starten →