Affect-aware conversational adaptation in mixed reality procedural tasks

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Abstract Conversational agents based on Large Language Models (LLMs) are increasingly explored in eXtended Reality (XR) applications that require real-time guidance during task execution. In task-oriented XR environments, conversational support must remain responsive and integrated with the ongoing activity, minimizing interruptions in user interaction. Although affect-aware adaptation has been proposed as a possible strategy to improve interaction quality, its impact in immersive task-oriented scenarios remains unclear. This paper presents a mixed-reality conversational system that integrates voice interaction, LLM-driven dialogue, and prosody-based affect-aware adaptation. The system adopts a modular client–server architecture with parallel semantic and affective processing pipelines, enabling emotion-related cues inferred from vocal prosody to be incorporated without interrupting conversational flow. The system was evaluated through a between-subjects study involving 40 participants performing a guided chemical procedure within a mixed-reality laboratory scenario. Participants interacted with either an affect-adaptive or a non-adaptive version of the conversational agent. The evaluation combined standardized questionnaires, behavioral metrics extracted from interaction logs, and qualitative feedback. Results showed that the affect-adaptive condition produced greater perceived usability scores compared to the non-adaptive condition. Participants interacting with the adaptive agent also required longer task completion times and engaged in a higher number of conversational turns. No significant differences emerged for overall workload between the two conditions. Qualitative findings further suggested that affect-aware adaptation made the interaction feel more natural and supportive.