EUVIMEDObjevte EUVIMED
Video/Audio

Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration

Wenyue Gong, Xiang Chen

Journal of Visualized Experiments · 2026

DOI: 10.3791/69830 ↗

At a glance

Video/Audio in Journal of Visualized Experiments, 2026. Written by Wenyue Gong and 1 co-authors. Institution: School of Design, Jiangnan University. Major MEDLINE topics: Artificial Intelligence. 40 references.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Achieving effective and natural human-AI collaboration remains a major challenge, particularly in dynamic real-world environments. The existing frameworks are often limited by the rigid single-modal inputs or rule-based multimodal fusion, restricting their robustness, personalization, and adaptability. This study introduces an adaptive multimodal interaction framework that integrates vision-based pose estimation and speech-based command understanding within a hierarchical human intention forecasting model. The framework employs Transformer-based sequence models for action prediction and Dynamic Time Warping to align human behavior with the structured task graphs for long-term planning. Unlike prior approaches that rely on static modality switching, our architecture applies an attention-based fusion mechanism to dynamically weight the most informative inputs. Furthermore, an online adaptation module enables real-time adjustment to user behavior, complemented by a lightweight explainability layer to foster user trust. The experimental evaluation in a collaborative assembly task demonstrates significant gains in task performance (up to 24%), intention prediction accuracy (up to 18%), and user satisfaction. These results highlight the effectiveness and versatility of the adaptive multimodal interaction frameworks, supporting their potential to advance collaborative intelligence toward more reliable, transparent, and human-oriented AI systems.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceHumansIntelligent SystemsCooperative BehaviorAdaptive Algorithms

Authors and institutions

  • School of Design, Jiangnan University.
  • School of Design, Jiangnan University; chenxiangsj@jiangnan.edu.cn.

Cite

APA

Gong, W., & Chen, X. (2026). Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. https://doi.org/10.3791/69830

Vancouver

Gong W, Chen X. Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. 2026. doi:10.3791/69830

BibTeX
@article{Gong2026,
  author = {Gong, Wenyue and Chen, Xiang},
  title = {Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration},
  journal = {Journal of Visualized Experiments},
  year = {2026},
  doi = {10.3791/69830}
}
Start for free →