EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction

Wing Yin Ng, Wanyu Ma, Pheng Ann Heng, Philip Wai Yan Chiu, Zheng Li

Advanced Intelligent Systems · 2026-01 · 8 (1)

DOI: 10.1002/aisy.202500483 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Advanced Intelligent Systems, online erschienen am 20.07.2025. Verfasst von Wing Yin Ng und 4 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Scrub nurses have crucial responsibilities, particularly in handling instrument‐related tasks. However, significant mental burdens and unfamiliarity with instruments can lead to various human errors. Consequently, the research community has explored robotic prototypes. Unfortunately, these prototypes often focus on specific instrument‐handling tasks or offer non‐intuitive interaction methods, hindering social acceptance. This article proposes a surgeon‐friendly robotic scrub nurse platform that addresses multiple instrument‐related tasks, including grasping and transferring, automatic sorting, and counting. To the best of the authors’ knowledge, this is the first prototype to incorporate audiovisual input modalities and a large language model (LLM) for smooth and intuitive interaction between the surgeon and the robot. Specifically, vision artificial intelligence (AI) provides accurate instrument detection results using oriented bounding boxes with an average precision of 97.6%, guiding robot motion planning. The speech AI recognizes the surgeon's voice commands. The LLM further interprets multimodal information to trigger different robot actions via the “tool use” capability, achieving a standalone success rate of 94% with an average action latency of less than 1 s on a real robotic scrub nurse hardware platform. Physical validation demonstrated that the proposed prototype successfully completed all assigned tasks, proving its feasibility and effectiveness.

Zitieren

APA

Ng, W. Y., Ma, W., Heng, P. A., Chiu, P. W. Y., & Li, Z. (2026). Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction. Advanced Intelligent Systems, 8(1). https://doi.org/10.1002/aisy.202500483

Vancouver

Ng WY, Ma W, Heng PA, Chiu PWY, Li Z. Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction. Advanced Intelligent Systems. 2026;8(1). doi:10.1002/aisy.202500483

BibTeX
@article{Ng2026,
  author = {Ng, Wing Yin and Ma, Wanyu and Heng, Pheng Ann and Chiu, Philip Wai Yan and Li, Zheng},
  title = {Large Language Model‐Embedded Intelligent Robotic Scrub Nurse with Multimodal Input for Enhancing Surgeon–Robot Interaction},
  journal = {Advanced Intelligent Systems},
  year = {2026},
  volume = {8},
  number = {1},
  doi = {10.1002/aisy.202500483}
}
Kostenlos starten →