EUVIMEDEUVIMED entdecken
Crossref

EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs

Zhen Fan, Peng Dai, Zhuo Su, Xu Gao, Zheng Lv, Jiarui Zhang, Tianyuan Du, Guidong Wang, Yang Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (3) · 2879-2887

DOI: 10.1609/aaai.v39i3.32294 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Zhen Fan und 8 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Egocentric human pose estimation (HPE) using wearable sensors is essential for VR/AR applications. Most methods rely solely on either egocentric-view images or sparse Inertial Measurement Unit (IMU) signals, leading to inaccuracies due to self-occlusion in images or the sparseness and drift of inertial sensors. Most importantly, the lack of real-world datasets containing both modalities is a major obstacle to progress in this field. To overcome the barrier, we propose EMHI, a multimodal Egocentric human Motion dataset with Head-Mounted Display (HMD) and body-worn IMUs, with all data collected under the real VR product suite. Specifically, EMHI provides synchronized stereo images from downward-sloping cameras on the headset and IMU data from body-worn sensors, along with pose annotations in SMPL format. This dataset consists of 885 sequences captured by 58 subjects performing 39 actions, totaling about 28.5 hours of recording. We evaluate the annotations by comparing them with optical marker-based SMPL fitting results. To substantiate the reliability of our dataset, we introduce MEPoser, a new baseline method for multimodal egocentric HPE, which employs a multimodal fusion encoder, temporal feature encoder, and MLP-based regression heads. The experiments on EMHI show that MEPoser outperforms existing single-modal methods and demonstrates the value of our dataset in solving the problem of egocentric HPE. We believe the release of EMHI and the method could advance the research of egocentric HPE and expedite the practical implementation of this technology in VR/AR products.

Zitieren

APA

Fan, Z., Dai, P., Su, Z., Gao, X., Lv, Z., Zhang, J., Du, T., Wang, G., & Zhang, Y. (2025). EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs. Proceedings of the AAAI Conference on Artificial Intelligence, 39(3), 2879-2887. https://doi.org/10.1609/aaai.v39i3.32294

Vancouver

Fan Z, Dai P, Su Z, Gao X, Lv Z, Zhang J, et al. EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(3):2879-2887. doi:10.1609/aaai.v39i3.32294

BibTeX
@article{Fan2025,
  author = {Fan, Zhen and Dai, Peng and Su, Zhuo and Gao, Xu and Lv, Zheng and Zhang, Jiarui and Du, Tianyuan and Wang, Guidong and Zhang, Yang},
  title = {EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {3},
  pages = {2879--2887},
  doi = {10.1609/aaai.v39i3.32294}
}
Kostenlos starten →