Advancing surgical VQA with scene graph knowledge
International Journal of Computer Assisted Radiology and Surgery · 23.05.2024 · 19 (7) · 1409-1417
DOI: 10.1007/s11548-024-03141-y ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, online erschienen am 23.05.2024. Verfasst von Kun Yuan und 5 weiteren Autor:innen. Beteiligte Einrichtung: University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France. kyuan@unistra.fr. Hauptthemen laut MEDLINE: Operating Rooms. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Agence Nationale de la Recherche.
Abstract (Originalsprache)
Purpose
The modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with natural language capabilities is emerging as a necessity. Our work aims to advance visual question answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question–condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design.
Methods
First, we propose a surgical scene graph-based dataset, SSG-VQA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. We then propose SSG-VQA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module, which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features.
Results
Our comprehensive analysis shows that our SSG-VQA dataset provides a more complex, diverse, geometrically grounded, unbiased and surgical action-oriented dataset compared to existing surgical VQA datasets and SSG-VQA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries.
Conclusion
We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. We point out that the bottleneck of the current surgical visual question–answer model lies in learning the encoded representation rather than decoding the sequence. Our SSG-VQA dataset provides a diagnostic benchmark to test the scene understanding and reasoning capabilities of the model. The source code and the dataset will be made publicly available at: https://github.com/CAMMA-public/SSG-VQA .
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France. kyuan@unistra.fr.
- IHU, Strasbourg, France. kyuan@unistra.fr.
- CAMP, Technische Universität München, Munich, Germany. kyuan@unistra.fr.
- University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France.
- IHU, Strasbourg, France.
- CAMP, Technische Universität München, Munich, Germany.
Förderung
- Agence Nationale de la Recherche
Zitieren
APA
Yuan, K., Kattel, M., Lavanchy, J. L., Navab, N., Srivastav, V., & Padoy, N. (2024). Advancing surgical VQA with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery, 19(7), 1409-1417. https://doi.org/10.1007/s11548-024-03141-y
Vancouver
Yuan K, Kattel M, Lavanchy JL, Navab N, Srivastav V, Padoy N. Advancing surgical VQA with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery. 2024;19(7):1409-1417. doi:10.1007/s11548-024-03141-y
BibTeX
@article{Yuan2024,
author = {Yuan, Kun and Kattel, Manasi and Lavanchy, Joël L and Navab, Nassir and Srivastav, Vinkle and Padoy, Nicolas},
title = {Advancing surgical VQA with scene graph knowledge},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2024},
volume = {19},
number = {7},
pages = {1409--1417},
doi = {10.1007/s11548-024-03141-y}
}
EUVIMED