EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution

Anjali Khurana, Amy Karlson, Christopher Collins, Mengjie Yu, Vinay Jayaram, Hrvoje Benko, Parmit K. Chilana

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 15.06.2026 · 10 (2) · 1-31

DOI: 10.1145/3810212 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 15.06.2026. Verfasst von Anjali Khurana und 6 weiteren Autor:innen. Die Arbeit nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

End-users of immersive Augmented Reality (AR) applications struggle to discover and execute unfamiliar mid-air gestures. We introduce ARG esture A id , a novel proof-of-concept conversational assistant that provides adaptive, context-aware assistance tailored to users' tasks and gestural errors. By combining users' voice-based task descriptions with formalized representations of hand gesture data together with LLMs, ARG esture A id dynamically supports users through: (1) feedforward , directing users toward correct pose, position, and placement before initiating gestures, and (2) corrective feedback , offering targeted, adaptive instructions to address user errors during discovery and execution. Our initial feasibility study (N=15), which compared users' real-world interaction data against a manually labeled reference dataset, shows that ARG esture A id achieved an overall accuracy of 81.3%. Our follow-up user study (N=15) evaluated ARG esture A id against a BASELINE LLM-based voice assistant across unimanual and bimanual gestures. Results indicate that ARG esture A id achieved higher task completion rates and was rated more favorably for perceived discoverability and usefulness. These findings suggest that adaptive, in-context feedforward and corrective feedback, grounded in formal gesture representations and observed error patterns, may support users during mid-air gesture discovery and execution, and help them refine their understanding of required gestures. We discuss key considerations for designing future adaptive, context-aware AR systems in AR environments.

Zitieren

APA

Khurana, A., Karlson, A., Collins, C., Yu, M., Jayaram, V., Benko, H., & Chilana, P. K. (2026). ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 10(2), 1-31. https://doi.org/10.1145/3810212

Vancouver

Khurana A, Karlson A, Collins C, Yu M, Jayaram V, Benko H, et al. ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2026;10(2):1-31. doi:10.1145/3810212

BibTeX
@article{Khurana2026,
  author = {Khurana, Anjali and Karlson, Amy and Collins, Christopher and Yu, Mengjie and Jayaram, Vinay and Benko, Hrvoje and Chilana, Parmit K.},
  title = {ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution},
  journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
  year = {2026},
  volume = {10},
  number = {2},
  pages = {1--31},
  doi = {10.1145/3810212}
}
Kostenlos starten →