LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access
IEEE Computer Graphics and Applications · 2026 · 1-14
DOI: 10.1109/mcg.2026.3701906 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Computer Graphics and Applications, 2026. Verfasst von Luca Cordioli und 2 weiteren Autor:innen.
Abstract (Originalsprache)
Extended Reality (XR) and Large Language Models (LLMs) can support context-aware, hands-free information access in industrial environments. This paper presents LlymX, a prototype combining a Unity-based 3D environment with an LLM-driven backend leveraging retrieval-augmented generation (RAG) and vector databases. LlymX introduces an interaction-oriented XR-LLM pipeline in which spatial user actions, such as gestures and object selection, are grounded in semantic object representations and linked to retrieval from a structured knowledge base. Through speech, gestures, and text, users can identify objects and access contextually relevant information grounded in domain documentation. A user study comparing LlymX against menu-based and infographic-based XR interfaces shows that the LLM-based modality is significantly more usable (SUS) than static infographics and comparable to traditional menus. Task completion time and cognitive workload showed no statistically significant differences, though participants appreciated the flexibility of conversational interaction.
Zitieren
APA
Cordioli, L., Valoriani, M., & Matera, M. (2026). LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access. IEEE Computer Graphics and Applications, 1-14. https://doi.org/10.1109/mcg.2026.3701906
Vancouver
Cordioli L, Valoriani M, Matera M. LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access. IEEE Computer Graphics and Applications. 2026:1-14. doi:10.1109/mcg.2026.3701906
BibTeX
@article{Cordioli2026,
author = {Cordioli, Luca and Valoriani, Matteo and Matera, Maristella},
title = {LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access},
journal = {IEEE Computer Graphics and Applications},
year = {2026},
pages = {1--14},
doi = {10.1109/mcg.2026.3701906}
}
EUVIMED