EUVIMEDEUVIMED entdecken
Crossref

SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation

Chen Lin, Zhenfeng Zhu, Yawei Zhao, Ying Zhang, Kunlun He, Yao Zhao

IEEE Transactions on Medical Imaging · 2024-04 · 43 (4) · 1337-1346

DOI: 10.1109/tmi.2023.3335909 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2024-04. Verfasst von Chen Lin und 5 weiteren Autor:innen. Hauptthemen laut MEDLINE: Semantics, Surgeons. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: Science and Technology Innovation 2030—New Generation Artificial Intelligence Major Project, National Natural Science Foundation of China, Beijing Natural Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automatically recording surgical procedures and generating surgical reports are crucial for alleviating surgeons' workload and enabling them to concentrate more on the operations. Despite some achievements, there still exist several issues for the previous works: 1) failure to model the interactive relationship between surgical instruments and tissue; and 2) neglect of fine-grained differences within different surgical images in the same surgery. To address these two issues, we propose an improved scene graph-guided Transformer, also named by SGT++, to generate more accurate surgical report, in which the complex interactions between surgical instruments and tissue are learnt from both explicit and implicit perspectives. Specifically, to facilitate the understanding of the surgical scene graph under a graph learning framework, a simple yet effective approach is proposed for homogenizing the input heterogeneous scene graph. For the homogeneous scene graph that contains explicit structured and fine-grained semantic relationships, we design an attention-induced graph transformer for node aggregation via an explicit relation-aware encoder. In addition, to characterize the implicit relationships about the instrument, tissue, and the interaction between them, the implicit relational attention is proposed to take full advantage of the prior knowledge from the interactional prototype memory. With the learnt explicit and implicit relation-aware representations, they are then coalesced to obtain the fused relation-aware representations contributing to generating reports. Some comprehensive experiments on two surgical datasets show that the proposed STG++ model achieves state-of-the-art results.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SemanticsSurgeonsHumans

Förderung

  • Science and Technology Innovation 2030—New Generation Artificial Intelligence Major Project
  • National Natural Science Foundation of China
  • Beijing Natural Science Foundation
  • National High Level Hospital Clinical Research Funding
  • Artificial Intelligence Public Service Platform for Screening and Auxiliary Diagnosis of Medical and Health Conditions

Zitieren

APA

Lin, C., Zhu, Z., Zhao, Y., Zhang, Y., He, K., & Zhao, Y. (2024). SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation. IEEE Transactions on Medical Imaging, 43(4), 1337-1346. https://doi.org/10.1109/tmi.2023.3335909

Vancouver

Lin C, Zhu Z, Zhao Y, Zhang Y, He K, Zhao Y. SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation. IEEE Transactions on Medical Imaging. 2024;43(4):1337-1346. doi:10.1109/tmi.2023.3335909

BibTeX
@article{Lin2024,
  author = {Lin, Chen and Zhu, Zhenfeng and Zhao, Yawei and Zhang, Ying and He, Kunlun and Zhao, Yao},
  title = {SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2024},
  volume = {43},
  number = {4},
  pages = {1337--1346},
  doi = {10.1109/tmi.2023.3335909}
}
Kostenlos starten →