EUVIMEDEUVIMED entdecken
Crossref

TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, Xiang Bai

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-05 · 48 (5) · 6008-6019

DOI: 10.1109/tpami.2026.3653415 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-05. Verfasst von Yuliang Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. Our approach introduces enhancement across several dimensions: By adopting Shifted Window Attention layer, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability. Evaluation on 12 benchmarks shows notable improvements: 5.2% in Scene Text-Centric tasks (including STVQA, TextVQA, and OCRVQA), 6.9% in Document-Oriented tasks (such as DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, and WikiTableQuestions), and 2.8% in Key Information Extraction tasks (comprising FUNSD, SROIE, and POIE). It outperforms in scene text spotting with a 10.9% increase and sets a new standard on OCRBench, a comprehensive benchmark consisting of 29 OCR-related assessments, with a score of 561, surpassing previous open-sourced large multimodal models for document understanding.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026). TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(5), 6008-6019. https://doi.org/10.1109/tpami.2026.3653415

Vancouver

Liu Y, Yang B, Liu Q, Li Z, Ma Z, Zhang S, et al. TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(5):6008-6019. doi:10.1109/tpami.2026.3653415

BibTeX
@article{Liu2026,
  author = {Liu, Yuliang and Yang, Biao and Liu, Qiang and Li, Zhang and Ma, Zhiyin and Zhang, Shuo and Bai, Xiang},
  title = {TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {5},
  pages = {6008--6019},
  doi = {10.1109/tpami.2026.3653415}
}
Kostenlos starten →