Augmented Reality-Based Training System Using Multimodal Language Model for Context-Aware Guidance and Activity Recognition in Complex Machine Operations
Designs · 05.03.2026 · 10 (2) · 30
DOI: 10.3390/designs10020030 ↗Kurz gefasst
Fachartikel in der Zeitschrift Designs, online erschienen am 05.03.2026. Verfasst von Waseem Ahmed und 1 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council (NSERC) of Canada, Graduate Enhancement of Tri-Council Stipends (GETS) program.
Abstract (Originalsprache)
Augmented Reality (AR) and Large Language Models (LLMs) have made significant advances across many fields, opening new possibilities, particularly in complex machine operations. In complex operations, non-expert users often struggle to perform high-precision tasks and require constant supervision to execute tasks correctly. This paper proposes a novel AR-MLLM-based training system that integrates AR, multimodal large language models (MLLMs), and prompt engineering to interpret real-time machine feedback and user activity. It converts extensive technical text into structured, step-by-step commands. The system uses a prompt structure developed through an iterative design method and refined across multiple machine operation scenarios, enabling ChatGPT to generate task-specific contextual digital overlays directly on the physical machines. A case study with participants was conducted to assess the effectiveness and usability of the AR-MLLM system in Coordinate Measuring Machine (CMM) operation training. The experimental results demonstrate high accuracy in task recognition and feature measurement activity. The data further show reduced time and user workload during task execution with the proposed AR-MLLM system. The proposed system not only provides real-time guidance and enhances efficiency in CMM operation training but also demonstrates the potential of the AR-MLLM design framework for broader industrial applications.
Förderung
- Natural Sciences and Engineering Research Council (NSERC) of Canada
- Graduate Enhancement of Tri-Council Stipends (GETS) program
Zitieren
APA
Ahmed, W., & Peng, Q. (2026). Augmented Reality-Based Training System Using Multimodal Language Model for Context-Aware Guidance and Activity Recognition in Complex Machine Operations. Designs, 10(2), 30. https://doi.org/10.3390/designs10020030
Vancouver
Ahmed W, Peng Q. Augmented Reality-Based Training System Using Multimodal Language Model for Context-Aware Guidance and Activity Recognition in Complex Machine Operations. Designs. 2026;10(2):30. doi:10.3390/designs10020030
BibTeX
@article{Ahmed2026,
author = {Ahmed, Waseem and Peng, Qingjin},
title = {Augmented Reality-Based Training System Using Multimodal Language Model for Context-Aware Guidance and Activity Recognition in Complex Machine Operations},
journal = {Designs},
year = {2026},
volume = {10},
number = {2},
pages = {30},
doi = {10.3390/designs10020030}
}
EUVIMED