SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation
IEEE Transactions on Medical Imaging · 2024-04 · 43 (4) · 1337-1346
DOI: 10.1109/tmi.2023.3335909 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2024-04. Verfasst von Chen Lin und 5 weiteren Autor:innen. Hauptthemen laut MEDLINE: Semantics, Surgeons. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: Science and Technology Innovation 2030—New Generation Artificial Intelligence Major Project, National Natural Science Foundation of China, Beijing Natural Science Foundation.
Abstract (Originalsprache)
Automatically recording surgical procedures and generating surgical reports are crucial for alleviating surgeons' workload and enabling them to concentrate more on the operations. Despite some achievements, there still exist several issues for the previous works: 1) failure to model the interactive relationship between surgical instruments and tissue; and 2) neglect of fine-grained differences within different surgical images in the same surgery. To address these two issues, we propose an improved scene graph-guided Transformer, also named by SGT++, to generate more accurate surgical report, in which the complex interactions between surgical instruments and tissue are learnt from both explicit and implicit perspectives. Specifically, to facilitate the understanding of the surgical scene graph under a graph learning framework, a simple yet effective approach is proposed for homogenizing the input heterogeneous scene graph. For the homogeneous scene graph that contains explicit structured and fine-grained semantic relationships, we design an attention-induced graph transformer for node aggregation via an explicit relation-aware encoder. In addition, to characterize the implicit relationships about the instrument, tissue, and the interaction between them, the implicit relational attention is proposed to take full advantage of the prior knowledge from the interactional prototype memory. With the learnt explicit and implicit relation-aware representations, they are then coalesced to obtain the fused relation-aware representations contributing to generating reports. Some comprehensive experiments on two surgical datasets show that the proposed STG++ model achieves state-of-the-art results.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- Science and Technology Innovation 2030—New Generation Artificial Intelligence Major Project
- National Natural Science Foundation of China
- Beijing Natural Science Foundation
- National High Level Hospital Clinical Research Funding
- Artificial Intelligence Public Service Platform for Screening and Auxiliary Diagnosis of Medical and Health Conditions
Zitieren
APA
Lin, C., Zhu, Z., Zhao, Y., Zhang, Y., He, K., & Zhao, Y. (2024). SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation. IEEE Transactions on Medical Imaging, 43(4), 1337-1346. https://doi.org/10.1109/tmi.2023.3335909
Vancouver
Lin C, Zhu Z, Zhao Y, Zhang Y, He K, Zhao Y. SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation. IEEE Transactions on Medical Imaging. 2024;43(4):1337-1346. doi:10.1109/tmi.2023.3335909
BibTeX
@article{Lin2024,
author = {Lin, Chen and Zhu, Zhenfeng and Zhao, Yawei and Zhang, Ying and He, Kunlun and Zhao, Yao},
title = {SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation},
journal = {IEEE Transactions on Medical Imaging},
year = {2024},
volume = {43},
number = {4},
pages = {1337--1346},
doi = {10.1109/tmi.2023.3335909}
}
EUVIMED