Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration
Journal of Visualized Experiments · 2026
DOI: 10.3791/69830 ↗Kurz gefasst
Video/Audio in der Zeitschrift Journal of Visualized Experiments, 2026. Verfasst von Wenyue Gong und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Design, Jiangnan University. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit nennt 40 Literaturangaben.
Abstract (Originalsprache)
Achieving effective and natural human-AI collaboration remains a major challenge, particularly in dynamic real-world environments. The existing frameworks are often limited by the rigid single-modal inputs or rule-based multimodal fusion, restricting their robustness, personalization, and adaptability. This study introduces an adaptive multimodal interaction framework that integrates vision-based pose estimation and speech-based command understanding within a hierarchical human intention forecasting model. The framework employs Transformer-based sequence models for action prediction and Dynamic Time Warping to align human behavior with the structured task graphs for long-term planning. Unlike prior approaches that rely on static modality switching, our architecture applies an attention-based fusion mechanism to dynamically weight the most informative inputs. Furthermore, an online adaptation module enables real-time adjustment to user behavior, complemented by a lightweight explainability layer to foster user trust. The experimental evaluation in a collaborative assembly task demonstrates significant gains in task performance (up to 24%), intention prediction accuracy (up to 18%), and user satisfaction. These results highlight the effectiveness and versatility of the adaptive multimodal interaction frameworks, supporting their potential to advance collaborative intelligence toward more reliable, transparent, and human-oriented AI systems.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- School of Design, Jiangnan University.
- School of Design, Jiangnan University; chenxiangsj@jiangnan.edu.cn.
Zitieren
APA
Gong, W., & Chen, X. (2026). Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. https://doi.org/10.3791/69830
Vancouver
Gong W, Chen X. Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. 2026. doi:10.3791/69830
BibTeX
@article{Gong2026,
author = {Gong, Wenyue and Chen, Xiang},
title = {Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration},
journal = {Journal of Visualized Experiments},
year = {2026},
doi = {10.3791/69830}
}
EUVIMED