EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection

Daria Hemmerling iD, Milosz Dudek iD, Justyna Krzywdziak, Magda Żbik iD, Wojciech Szecowka, Mateusz Daniol iD, Marek Wodzinski iD, Monika Rudzinska-Bar, Magdalena Wojcik-Pedziwiatr iD

Sensors · 2026

DOI: 10.3390/s26010269 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2026. Verfasst von Daria Hemmerling und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland. Hauptthemen laut MEDLINE: Parkinson Disease, Speech, Head. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: The National Centre for Research and Development, Poland.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Parkinson’s disease (PD) alters both speech and movement, yet most automated assessments still treat these signals separately. We examined whether combining voice with head motion improves discrimination between patients and healthy controls (HC). Synchronous measurements of acoustic and inertial signals were collected using a HoloLens 2 headset. Data were obtained from 165 participants (72 PD/93 HC), following a standardized mixed-reality (MR) protocol. We benchmarked single-modality models against fusion strategies under 5-fold stratified cross-validation. Voice alone was robust (pooled AUC ≈ 0.865), while the inertial channel alone was near chance (AUC ≈ 0.497). Fusion provided a modest but repeatable improvement: gated early-fusion achieved the highest AUC (≈0.875), cross-attention fusion was comparable (≈0.873). Gains were task-dependent. While speech-dominated tasks were already well captured by audio, tasks that embed movement benefited from complementary inertial data. Proposed MR capture proved feasible within a single session and showed that motion acts as a conditional improvement factor rather than a sole predictor. The results outline a practical path to multimodal screening and monitoring for PD, preserving the reliability of acoustic biomarkers while integrating kinematic features when they matter.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Parkinson DiseaseSpeechHeadHumansAgedMaleFemaleMiddle AgedAugmented RealityVoiceSignal Processing, Computer-AssistedHead MovementsAcoustics

Stichwörter der Autor:innen

Parkinson’s diseaseaugmented reality (AR)gyroscope and accelerometer signalshead motioninertial measurement units (IMU)multimodal data fusionremote digital assessmentspeech analysisvoice biomarkerswearable sensing

Autor:innen und Institutionen

  • Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland.
  • Department of Neurology, Andrzej Frycz Modrzewski Krakow University, 30-705 Krakow, Poland.

ORCID-Profile: Daria Hemmerling, Milosz Dudek, Magda Żbik, Mateusz Daniol, Marek Wodzinski, Magdalena Wojcik-Pedziwiatr

Förderung

  • The National Centre for Research and Development, Poland

Zitieren

APA

Hemmerling, D., Dudek, M., Krzywdziak, J., Żbik, M., Szecowka, W., Daniol, M., Wodzinski, M., Rudzinska-Bar, M., & Wojcik-Pedziwiatr, M. (2026). GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection. Sensors. https://doi.org/10.3390/s26010269

Vancouver

Hemmerling D, Dudek M, Krzywdziak J, Żbik M, Szecowka W, Daniol M, et al. GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection. Sensors. 2026. doi:10.3390/s26010269

BibTeX
@article{Hemmerling2026,
  author = {Hemmerling, Daria and Dudek, Milosz and Krzywdziak, Justyna and Żbik, Magda and Szecowka, Wojciech and Daniol, Mateusz and Wodzinski, Marek and Rudzinska-Bar, Monika and Wojcik-Pedziwiatr, Magdalena},
  title = {GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson’s Disease Detection},
  journal = {Sensors},
  year = {2026},
  doi = {10.3390/s26010269}
}
Kostenlos starten →