ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 15.06.2026 · 10 (2) · 1-31
DOI: 10.1145/3810212 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 15.06.2026. Verfasst von Anjali Khurana und 6 weiteren Autor:innen. Die Arbeit nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
End-users of immersive Augmented Reality (AR) applications struggle to discover and execute unfamiliar mid-air gestures. We introduce ARG esture A id , a novel proof-of-concept conversational assistant that provides adaptive, context-aware assistance tailored to users' tasks and gestural errors. By combining users' voice-based task descriptions with formalized representations of hand gesture data together with LLMs, ARG esture A id dynamically supports users through: (1) feedforward , directing users toward correct pose, position, and placement before initiating gestures, and (2) corrective feedback , offering targeted, adaptive instructions to address user errors during discovery and execution. Our initial feasibility study (N=15), which compared users' real-world interaction data against a manually labeled reference dataset, shows that ARG esture A id achieved an overall accuracy of 81.3%. Our follow-up user study (N=15) evaluated ARG esture A id against a BASELINE LLM-based voice assistant across unimanual and bimanual gestures. Results indicate that ARG esture A id achieved higher task completion rates and was rated more favorably for perceived discoverability and usefulness. These findings suggest that adaptive, in-context feedforward and corrective feedback, grounded in formal gesture representations and observed error patterns, may support users during mid-air gesture discovery and execution, and help them refine their understanding of required gestures. We discuss key considerations for designing future adaptive, context-aware AR systems in AR environments.
Zitieren
APA
Khurana, A., Karlson, A., Collins, C., Yu, M., Jayaram, V., Benko, H., & Chilana, P. K. (2026). ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 10(2), 1-31. https://doi.org/10.1145/3810212
Vancouver
Khurana A, Karlson A, Collins C, Yu M, Jayaram V, Benko H, et al. ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2026;10(2):1-31. doi:10.1145/3810212
BibTeX
@article{Khurana2026,
author = {Khurana, Anjali and Karlson, Amy and Collins, Christopher and Yu, Mengjie and Jayaram, Vinay and Benko, Hrvoje and Chilana, Parmit K.},
title = {ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
year = {2026},
volume = {10},
number = {2},
pages = {1--31},
doi = {10.1145/3810212}
}
EUVIMED