LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery
Advanced Intelligent Systems · 23.08.2026
DOI: 10.1002/aisy.70524 ↗Kurz gefasst
Fachartikel in der Zeitschrift Advanced Intelligent Systems, online erschienen am 23.08.2026. Verfasst von Yisen Huang und 7 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Minimally invasive surgery (MIS) has gained widespread adoption over open surgery due to its numerous benefits for patient outcomes. In laparoscopic surgical procedures, maintaining a stable and optimal camera view is crucial, making human endoscope holders indispensable members of the MIS team, as they ensure consistent visualization during operations. However, their performance can vary due to factors such as experience level and fatigue. As a result, robotic systems have emerged as promising alternatives, offering precision and stability. Nevertheless, existing solutions often suffer from nonintuitive interfaces or focus solely on isolated sub‐tasks within the control workflow, limiting their overall effectiveness. To address these challenges and advance the development of robotic endoscopes, we present a novel robotic endoscope control framework powered by a large language model (LLM). Our system integrates both manual and autonomous functionalities, employing a real‐time deep learning‐based instrument tip detection model for accurate visual tracking and an automatic speech recognition module for natural language command interpretation. The LLM‐based central logic unit processes multimodal inputs through tool use capability to trigger and control the endoscope's movements. Experimental results validate the system's practicality, achieving an 89.47% success rate with a response latency of ≈1 s. Related codes and LLM prompts will be made available in the GitHub repository.
Zitieren
APA
Huang, Y., Ng, W. Y., Dou, Q., Wu, X., Chan, S. M., Liu, Y., Chiu, P. W. Y., & Li, Z. (2026). LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery. Advanced Intelligent Systems. https://doi.org/10.1002/aisy.70524
Vancouver
Huang Y, Ng WY, Dou Q, Wu X, Chan SM, Liu Y, et al. LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery. Advanced Intelligent Systems. 2026. doi:10.1002/aisy.70524
BibTeX
@article{Huang2026,
author = {Huang, Yisen and Ng, Wing Yin and Dou, Qi and Wu, Xixin and Chan, Shannon Melissa and Liu, Yunhui and Chiu, Philip Wai Yan and Li, Zheng},
title = {LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery},
journal = {Advanced Intelligent Systems},
year = {2026},
doi = {10.1002/aisy.70524}
}
EUVIMED