A Multimodal Audiovisual Deep Learning Framework for Early Detection of Parkinson’s Disease
Electronics · 30.04.2026 · 15 (9) · 1904
DOI: 10.3390/electronics15091904 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 30.04.2026. Verfasst von Yinpeng Guo und 6 weiteren Autor:innen. Die Arbeit nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, the Major Science and Technology Program of Henan Province, the Central Government Guiding Local Science and Technology Development Fund Program of Henan Province.
Abstract (Originalsprache)
Parkinson’s disease (PD) is a progressive neurodegenerative disorder primarily caused by the degeneration of dopamine-producing neurons in the substantia nigra, leading to characteristic motor symptoms such as tremors, rigidity, and bradykinesia, as well as non-motor manifestations including depression, sleep disturbances, and speech impairments. Among these symptoms, speech abnormalities affect approximately 90% of individuals with PD, making acoustic analysis a promising non-invasive cue for early detection. However, subtle speech variations are often imperceptible to the human ear, and speech-only analysis may overlook complementary visual manifestations, such as hypomimia—reduced facial expressivity commonly observed in PD patients. To address these limitations, we propose Parkinson’s Detection via Attentional Fusion Network (PDAF-Net), a novel multimodal deep learning framework for early PD detection that jointly models acoustic and facial dynamic features in a binary classification setting. The proposed architecture consists of a Dual-Stream Feature Encoder (DSFE), with an audio branch based on a one-dimensional convolutional neural network (1D-CNN) and bidirectional long short-term memory (BiLSTM), and a visual branch built upon a two-dimensional convolutional neural network (2D-CNN) and a Transformer encoder. Multimodal integration is achieved through a Cross-Attention-guided Attentional Feature Fusion (CA-AFF) module, which explicitly models bidirectional cross-modal interactions and performs adaptive feature recalibration via an iterative attentional fusion mechanism. We conducted experiments on a self-collected Chinese multimodal dataset comprising 100 PD patients and 100 healthy controls. Although the data are balanced at the subject level, sliding-window segmentation introduces sample-level imbalance; to address this issue, a class-balanced focal loss is employed. Model performance was evaluated using subject-wise five-fold cross-validation. The results demonstrate that PDAF-Net consistently outperforms unimodal baselines across multiple evaluation metrics, achieving an accuracy of 89.3%, an F1-score of 0.884, and an AUC of 0.916. These findings highlight the effectiveness of explicit cross-modal interaction modeling and adaptive feature fusion for improving automated early PD screening in real-world clinical settings.
Förderung
- National Natural Science Foundation of China
- the Major Science and Technology Program of Henan Province
- the Central Government Guiding Local Science and Technology Development Fund Program of Henan Province
Zitieren
APA
Guo, Y., Huo, H., Pei, Y., Ma, L., Kang, S., Xu, J., & Mei, A. (2026). A Multimodal Audiovisual Deep Learning Framework for Early Detection of Parkinson’s Disease. Electronics, 15(9), 1904. https://doi.org/10.3390/electronics15091904
Vancouver
Guo Y, Huo H, Pei Y, Ma L, Kang S, Xu J, et al. A Multimodal Audiovisual Deep Learning Framework for Early Detection of Parkinson’s Disease. Electronics. 2026;15(9):1904. doi:10.3390/electronics15091904
BibTeX
@article{Guo2026,
author = {Guo, Yinpeng and Huo, Hua and Pei, Yulong and Ma, Lan and Kang, Shilu and Xu, Jiaxin and Mei, Aokun},
title = {A Multimodal Audiovisual Deep Learning Framework for Early Detection of Parkinson’s Disease},
journal = {Electronics},
year = {2026},
volume = {15},
number = {9},
pages = {1904},
doi = {10.3390/electronics15091904}
}
EUVIMED