EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery

Yisen Huang, Wing Yin Ng, Qi Dou, Xixin Wu, Shannon Melissa Chan, Yunhui Liu, Philip Wai Yan Chiu, Zheng Li

Advanced Intelligent Systems · 23.08.2026

DOI: 10.1002/aisy.70524 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Advanced Intelligent Systems, online erschienen am 23.08.2026. Verfasst von Yisen Huang und 7 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Minimally invasive surgery (MIS) has gained widespread adoption over open surgery due to its numerous benefits for patient outcomes. In laparoscopic surgical procedures, maintaining a stable and optimal camera view is crucial, making human endoscope holders indispensable members of the MIS team, as they ensure consistent visualization during operations. However, their performance can vary due to factors such as experience level and fatigue. As a result, robotic systems have emerged as promising alternatives, offering precision and stability. Nevertheless, existing solutions often suffer from nonintuitive interfaces or focus solely on isolated sub‐tasks within the control workflow, limiting their overall effectiveness. To address these challenges and advance the development of robotic endoscopes, we present a novel robotic endoscope control framework powered by a large language model (LLM). Our system integrates both manual and autonomous functionalities, employing a real‐time deep learning‐based instrument tip detection model for accurate visual tracking and an automatic speech recognition module for natural language command interpretation. The LLM‐based central logic unit processes multimodal inputs through tool use capability to trigger and control the endoscope's movements. Experimental results validate the system's practicality, achieving an 89.47% success rate with a response latency of ≈1 s. Related codes and LLM prompts will be made available in the GitHub repository.

Zitieren

APA

Huang, Y., Ng, W. Y., Dou, Q., Wu, X., Chan, S. M., Liu, Y., Chiu, P. W. Y., & Li, Z. (2026). LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery. Advanced Intelligent Systems. https://doi.org/10.1002/aisy.70524

Vancouver

Huang Y, Ng WY, Dou Q, Wu X, Chan SM, Liu Y, et al. LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery. Advanced Intelligent Systems. 2026. doi:10.1002/aisy.70524

BibTeX
@article{Huang2026,
  author = {Huang, Yisen and Ng, Wing Yin and Dou, Qi and Wu, Xixin and Chan, Shannon Melissa and Liu, Yunhui and Chiu, Philip Wai Yan and Li, Zheng},
  title = {LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery},
  journal = {Advanced Intelligent Systems},
  year = {2026},
  doi = {10.1002/aisy.70524}
}
Kostenlos starten →