GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection
Sensors · 2026
DOI: 10.3390/s26010269 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, 2026. Verfasst von Daria Hemmerling und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland. Hauptthemen laut MEDLINE: Parkinson Disease, Speech, Head. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: The National Centre for Research and Development, Poland.
Abstract (Originalsprache)
Parkinson’s disease (PD) alters both speech and movement, yet most automated assessments still treat these signals separately. We examined whether combining voice with head motion improves discrimination between patients and healthy controls (HC). Synchronous measurements of acoustic and inertial signals were collected using a HoloLens 2 headset. Data were obtained from 165 participants (72 PD/93 HC), following a standardized mixed-reality (MR) protocol. We benchmarked single-modality models against fusion strategies under 5-fold stratified cross-validation. Voice alone was robust (pooled AUC ≈ 0.865), while the inertial channel alone was near chance (AUC ≈ 0.497). Fusion provided a modest but repeatable improvement: gated early-fusion achieved the highest AUC (≈0.875), cross-attention fusion was comparable (≈0.873). Gains were task-dependent. While speech-dominated tasks were already well captured by audio, tasks that embed movement benefited from complementary inertial data. Proposed MR capture proved feasible within a single session and showed that motion acts as a conditional improvement factor rather than a sole predictor. The results outline a practical path to multimodal screening and monitoring for PD, preserving the reliability of acoustic biomarkers while integrating kinematic features when they matter.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland.
- Department of Neurology, Andrzej Frycz Modrzewski Krakow University, 30-705 Krakow, Poland.
ORCID-Profile: Daria Hemmerling, Milosz Dudek, Magda Żbik, Mateusz Daniol, Marek Wodzinski, Magdalena Wojcik-Pedziwiatr
Förderung
- The National Centre for Research and Development, Poland
Zitieren
APA
Hemmerling, D., Dudek, M., Krzywdziak, J., Żbik, M., Szecowka, W., Daniol, M., Wodzinski, M., Rudzinska-Bar, M., & Wojcik-Pedziwiatr, M. (2026). GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection. Sensors. https://doi.org/10.3390/s26010269
Vancouver
Hemmerling D, Dudek M, Krzywdziak J, Żbik M, Szecowka W, Daniol M, et al. GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection. Sensors. 2026. doi:10.3390/s26010269
BibTeX
@article{Hemmerling2026,
author = {Hemmerling, Daria and Dudek, Milosz and Krzywdziak, Justyna and Żbik, Magda and Szecowka, Wojciech and Daniol, Mateusz and Wodzinski, Marek and Rudzinska-Bar, Monika and Wojcik-Pedziwiatr, Magdalena},
title = {GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection},
journal = {Sensors},
year = {2026},
doi = {10.3390/s26010269}
}
EUVIMED