EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model

Jiquan Liu iD, Yichen Zhu, Jingyi Feng iD, Xiaoyan Zhang, Ziyu Zhou, Ye Tao, Huilong Duan

PLOS One · 17.03.2026 · 21 (3) · e0343823

DOI: 10.1371/journal.pone.0343823 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 17.03.2026. Verfasst von Jiquan Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Ha…. Hauptthemen laut MEDLINE: Information Storage and Retrieval. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, Zhejiang Key Research and Development Program of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Surgical image captioning is critical for automated reporting and education but is currently limited by a lack of long-text datasets and the tendency of generic Multimodal Large Language Models (MLLMs) to hallucinate medical details. To address this, we present a comprehensive framework for long-text surgical captioning. First, we construct a verified long-text benchmark extending the EndoVis2018 dataset, utilizing an automated pipeline with expert-in-the-loop validation to transform brief triplets into rich narratives. Second, we investigate domain-specific adaptation strategies for MLLMs. We implement a surgical concept retrieval-augmented generation (RAG) mechanism that dynamically injects specialized knowledge (instruments, actions) into the visual encoder, effectively mitigating domain-specific hallucinations common in generic models. Finally, recognizing the inadequacy of n-gram metrics for long medical text, we establish a robust evaluation protocol using clinically-aligned metrics. Extensive experiments demonstrate that our data-centric and retrieval-enhanced approach significantly outperforms baselines in producing clinically accurate, coherent long descriptions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Information Storage and RetrievalLarge Language ModelsHumans

Autor:innen und Institutionen

  • Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China.
  • The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China.

ORCID-Profile: Jiquan Liu, Jingyi Feng

Förderung

  • National Key Research and Development Program of China
  • Zhejiang Key Research and Development Program of China

Zitieren

APA

Liu, J., Zhu, Y., Feng, J., Zhang, X., Zhou, Z., Tao, Y., & Duan, H. (2026). Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model. PLOS One, 21(3), e0343823. https://doi.org/10.1371/journal.pone.0343823

Vancouver

Liu J, Zhu Y, Feng J, Zhang X, Zhou Z, Tao Y, et al. Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model. PLOS One. 2026;21(3):e0343823. doi:10.1371/journal.pone.0343823

BibTeX
@article{Liu2026,
  author = {Liu, Jiquan and Zhu, Yichen and Feng, Jingyi and Zhang, Xiaoyan and Zhou, Ziyu and Tao, Ye and Duan, Huilong},
  title = {Long-text caption generation for surgical image with a concept retrieval augmented large multimodal model},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {3},
  pages = {e0343823},
  doi = {10.1371/journal.pone.0343823}
}
Kostenlos starten →