A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly
IET Collaborative Intelligent Manufacturing · 2026-01 · 8 (1)
DOI: 10.1049/cim2.70086 ↗Kurz gefasst
Fachartikel in der Zeitschrift IET Collaborative Intelligent Manufacturing, online erschienen am 01.11.2026. Verfasst von Rui Li und 7 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
With the development of flexible manufacturing and smart factories, human–robot collaborative assembly is moving towards large‐scale deployment. However, lighting variations, occlusions, and the complex motion characteristics of collaborative assembly actions often challenge reliable perception and fine‐grained action recognition. To address these challenges, this paper proposes a task‐oriented multi‐view and multimodal framework for collaborative assembly action recognition. First, an assembly‐oriented whole‐body pose estimator termed HigherPose is developed to emphasise manipulation‐related keypoints and improve pose localization under complex industrial conditions. On COCO‐WholeBody, HigherPose reaches 67.1 AP on whole‐body keypoints, exceeding ZoomNAS by 1.7 AP while reducing computation by 0.5 GFLOPs. Furthermore, a viewpoint–modality coordination network, termed MMARN (multi‐view multimodal action recognition network), is designed to exploit complementary workspace‐context and human‐motion information from red–green–blue (RGB) observations and skeleton sequences. Evaluated on the industrial collaborative assembly dataset InHard65, MMARN achieves the best validation performance among the evaluated configurations, with 99.39% Top‐1 accuracy, and qualitative results show reliable discrimination of fine‐grained operational intentions. Overall, the results demonstrate the effectiveness of the proposed task‐oriented two‐stage framework for fine‐grained human–robot collaborative assembly action recognition.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Li, R., Wu, J., Yang, S., Liu, X., Li, H., Cai, Y., Liu, Z., & Liu, Y. (2026). A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly. IET Collaborative Intelligent Manufacturing, 8(1). https://doi.org/10.1049/cim2.70086
Vancouver
Li R, Wu J, Yang S, Liu X, Li H, Cai Y, et al. A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly. IET Collaborative Intelligent Manufacturing. 2026;8(1). doi:10.1049/cim2.70086
BibTeX
@article{Li2026,
author = {Li, Rui and Wu, Jiaxin and Yang, Shiqiang and Liu, Xutao and Li, Huandong and Cai, Yuanyuan and Liu, Zhuo and Liu, Yang},
title = {A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly},
journal = {IET Collaborative Intelligent Manufacturing},
year = {2026},
volume = {8},
number = {1},
doi = {10.1049/cim2.70086}
}
EUVIMED