EUVIMEDEUVIMED entdecken
Video/Audio

Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration

Wenyue Gong, Xiang Chen

Journal of Visualized Experiments · 2026

DOI: 10.3791/69830 ↗

Kurz gefasst

Video/Audio in der Zeitschrift Journal of Visualized Experiments, 2026. Verfasst von Wenyue Gong und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Design, Jiangnan University. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit nennt 40 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Achieving effective and natural human-AI collaboration remains a major challenge, particularly in dynamic real-world environments. The existing frameworks are often limited by the rigid single-modal inputs or rule-based multimodal fusion, restricting their robustness, personalization, and adaptability. This study introduces an adaptive multimodal interaction framework that integrates vision-based pose estimation and speech-based command understanding within a hierarchical human intention forecasting model. The framework employs Transformer-based sequence models for action prediction and Dynamic Time Warping to align human behavior with the structured task graphs for long-term planning. Unlike prior approaches that rely on static modality switching, our architecture applies an attention-based fusion mechanism to dynamically weight the most informative inputs. Furthermore, an online adaptation module enables real-time adjustment to user behavior, complemented by a lightweight explainability layer to foster user trust. The experimental evaluation in a collaborative assembly task demonstrates significant gains in task performance (up to 24%), intention prediction accuracy (up to 18%), and user satisfaction. These results highlight the effectiveness and versatility of the adaptive multimodal interaction frameworks, supporting their potential to advance collaborative intelligence toward more reliable, transparent, and human-oriented AI systems.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceHumansIntelligent SystemsCooperative BehaviorAdaptive Algorithms

Autor:innen und Institutionen

  • School of Design, Jiangnan University.
  • School of Design, Jiangnan University; chenxiangsj@jiangnan.edu.cn.

Zitieren

APA

Gong, W., & Chen, X. (2026). Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. https://doi.org/10.3791/69830

Vancouver

Gong W, Chen X. Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration. Journal of Visualized Experiments. 2026. doi:10.3791/69830

BibTeX
@article{Gong2026,
  author = {Gong, Wenyue and Chen, Xiang},
  title = {Designing Adaptive Multimodal Interaction Frameworks to Enhance Human-AI Collaboration},
  journal = {Journal of Visualized Experiments},
  year = {2026},
  doi = {10.3791/69830}
}
Kostenlos starten →