Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction
Advanced Intelligent Systems · 2026-01 · 8 (1)
DOI: 10.1002/aisy.202500483 ↗Kurz gefasst
Fachartikel in der Zeitschrift Advanced Intelligent Systems, online erschienen am 20.07.2025. Verfasst von Wing Yin Ng und 4 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Scrub nurses have crucial responsibilities, particularly in handling instrument‐related tasks. However, significant mental burdens and unfamiliarity with instruments can lead to various human errors. Consequently, the research community has explored robotic prototypes. Unfortunately, these prototypes often focus on specific instrument‐handling tasks or offer non‐intuitive interaction methods, hindering social acceptance. This article proposes a surgeon‐friendly robotic scrub nurse platform that addresses multiple instrument‐related tasks, including grasping and transferring, automatic sorting, and counting. To the best of the authors’ knowledge, this is the first prototype to incorporate audiovisual input modalities and a large language model (LLM) for smooth and intuitive interaction between the surgeon and the robot. Specifically, vision artificial intelligence (AI) provides accurate instrument detection results using oriented bounding boxes with an average precision of 97.6%, guiding robot motion planning. The speech AI recognizes the surgeon's voice commands. The LLM further interprets multimodal information to trigger different robot actions via the “tool use” capability, achieving a standalone success rate of 94% with an average action latency of less than 1 s on a real robotic scrub nurse hardware platform. Physical validation demonstrated that the proposed prototype successfully completed all assigned tasks, proving its feasibility and effectiveness.
Zitieren
APA
Ng, W. Y., Ma, W., Heng, P. A., Chiu, P. W. Y., & Li, Z. (2026). Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction. Advanced Intelligent Systems, 8(1). https://doi.org/10.1002/aisy.202500483
Vancouver
Ng WY, Ma W, Heng PA, Chiu PWY, Li Z. Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction. Advanced Intelligent Systems. 2026;8(1). doi:10.1002/aisy.202500483
BibTeX
@article{Ng2026,
author = {Ng, Wing Yin and Ma, Wanyu and Heng, Pheng Ann and Chiu, Philip Wai Yan and Li, Zheng},
title = {Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction},
journal = {Advanced Intelligent Systems},
year = {2026},
volume = {8},
number = {1},
doi = {10.1002/aisy.202500483}
}
EUVIMED