EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly

Rui Li, Jiaxin Wu, Shiqiang Yang, Xutao Liu, Huandong Li, Yuanyuan Cai, Zhuo Liu, Yang Liu

IET Collaborative Intelligent Manufacturing · 2026-01 · 8 (1)

DOI: 10.1049/cim2.70086 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IET Collaborative Intelligent Manufacturing, online erschienen am 01.11.2026. Verfasst von Rui Li und 7 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the development of flexible manufacturing and smart factories, human–robot collaborative assembly is moving towards large‐scale deployment. However, lighting variations, occlusions, and the complex motion characteristics of collaborative assembly actions often challenge reliable perception and fine‐grained action recognition. To address these challenges, this paper proposes a task‐oriented multi‐view and multimodal framework for collaborative assembly action recognition. First, an assembly‐oriented whole‐body pose estimator termed HigherPose is developed to emphasise manipulation‐related keypoints and improve pose localization under complex industrial conditions. On COCO‐WholeBody, HigherPose reaches 67.1 AP on whole‐body keypoints, exceeding ZoomNAS by 1.7 AP while reducing computation by 0.5 GFLOPs. Furthermore, a viewpoint–modality coordination network, termed MMARN (multi‐view multimodal action recognition network), is designed to exploit complementary workspace‐context and human‐motion information from red–green–blue (RGB) observations and skeleton sequences. Evaluated on the industrial collaborative assembly dataset InHard65, MMARN achieves the best validation performance among the evaluated configurations, with 99.39% Top‐1 accuracy, and qualitative results show reliable discrimination of fine‐grained operational intentions. Overall, the results demonstrate the effectiveness of the proposed task‐oriented two‐stage framework for fine‐grained human–robot collaborative assembly action recognition.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Li, R., Wu, J., Yang, S., Liu, X., Li, H., Cai, Y., Liu, Z., & Liu, Y. (2026). A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly. IET Collaborative Intelligent Manufacturing, 8(1). https://doi.org/10.1049/cim2.70086

Vancouver

Li R, Wu J, Yang S, Liu X, Li H, Cai Y, et al. A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly. IET Collaborative Intelligent Manufacturing. 2026;8(1). doi:10.1049/cim2.70086

BibTeX
@article{Li2026,
  author = {Li, Rui and Wu, Jiaxin and Yang, Shiqiang and Liu, Xutao and Li, Huandong and Cai, Yuanyuan and Liu, Zhuo and Liu, Yang},
  title = {A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly},
  journal = {IET Collaborative Intelligent Manufacturing},
  year = {2026},
  volume = {8},
  number = {1},
  doi = {10.1049/cim2.70086}
}
Kostenlos starten →