EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data

Dayoung Jeong, Kyungsik Han

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 13.05.2024 · 8 (2) · 1-24

DOI: 10.1145/3659594 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 15.05.2024. Verfasst von Dayoung Jeong und 1 weiteren Autor:innen. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 95 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0). Gefördert von: The National Research Foundation, Institute of Information & communications Technology Planning & Evaluation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Cybersickness, a factor that hinders user immersion in VR, has been the subject of ongoing attempts to predict it using AI. Previous studies have used CNN and LSTM for prediction models and used attention mechanisms and XAI for data analysis, yet none explored a transformer that can better reflect the spatial and temporal characteristics of the data, beneficial for enhancing prediction and feature importance analysis. In this paper, we propose cybersickness prediction models using multimodal time-series sensor data (i.e., eye movement, head movement, and physiological signals) based on a transformer algorithm, considering sensor data pre-processing and multimodal data fusion methods. We constructed the MSCVR dataset consisting of normalized sensor data, spectrogram formatted sensor data, and cybersickness levels collected from 45 participants through a user study. We proposed two methods for embedding multimodal time-series sensor data into the transformer: modality-specific spatial and temporal transformer encoders for normalized sensor data (MS-STTN) and modality-specific spatial-temporal transformer encoder for spectrogram (MS-STTS). MS-STTN yielded the highest performance in the ablation study and the comparison of the existing models. Furthermore, by analyzing the importance of data features, we determined their relevance to cybersickness over time, especially the salience of eye movement features. Our results and insights derived from multimodal time-series sensor data and the transformer model provide a comprehensive understanding of cybersickness and its association with sensor data. Our MSCVR dataset and code are publicly available: https://github.com/dayoung-jeong/PRECYSE.git.

Förderung

  • The National Research Foundation
  • Institute of Information & communications Technology Planning & Evaluation

Zitieren

APA

Jeong, D., & Han, K. (2024). PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(2), 1-24. https://doi.org/10.1145/3659594

Vancouver

Jeong D, Han K. PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2024;8(2):1-24. doi:10.1145/3659594

BibTeX
@article{Jeong2024,
  author = {Jeong, Dayoung and Han, Kyungsik},
  title = {PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data},
  journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
  year = {2024},
  volume = {8},
  number = {2},
  pages = {1--24},
  doi = {10.1145/3659594}
}
Kostenlos starten →