Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model
PLOS One · 17.03.2026 · 21 (3) · e0343823
DOI: 10.1371/journal.pone.0343823 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 17.03.2026. Verfasst von Jiquan Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Ha…. Hauptthemen laut MEDLINE: Information Storage and Retrieval. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, Zhejiang Key Research and Development Program of China.
Abstract (Originalsprache)
Surgical image captioning is critical for automated reporting and education but is currently limited by a lack of long-text datasets and the tendency of generic Multimodal Large Language Models (MLLMs) to hallucinate medical details. To address this, we present a comprehensive framework for long-text surgical captioning. First, we construct a verified long-text benchmark extending the EndoVis2018 dataset, utilizing an automated pipeline with expert-in-the-loop validation to transform brief triplets into rich narratives. Second, we investigate domain-specific adaptation strategies for MLLMs. We implement a surgical concept retrieval-augmented generation (RAG) mechanism that dynamically injects specialized knowledge (instruments, actions) into the visual encoder, effectively mitigating domain-specific hallucinations common in generic models. Finally, recognizing the inadequacy of n-gram metrics for long medical text, we establish a robust evaluation protocol using clinically-aligned metrics. Extensive experiments demonstrate that our data-centric and retrieval-enhanced approach significantly outperforms baselines in producing clinically accurate, coherent long descriptions.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China.
- The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China.
ORCID-Profile: Jiquan Liu, Jingyi Feng
Förderung
- National Key Research and Development Program of China
- Zhejiang Key Research and Development Program of China
Zitieren
APA
Liu, J., Zhu, Y., Feng, J., Zhang, X., Zhou, Z., Tao, Y., & Duan, H. (2026). Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model. PLOS One, 21(3), e0343823. https://doi.org/10.1371/journal.pone.0343823
Vancouver
Liu J, Zhu Y, Feng J, Zhang X, Zhou Z, Tao Y, et al. Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model. PLOS One. 2026;21(3):e0343823. doi:10.1371/journal.pone.0343823
BibTeX
@article{Liu2026,
author = {Liu, Jiquan and Zhu, Yichen and Feng, Jingyi and Zhang, Xiaoyan and Zhou, Ziyu and Tao, Ye and Duan, Huilong},
title = {Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {3},
pages = {e0343823},
doi = {10.1371/journal.pone.0343823}
}
EUVIMED