Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders
Journal of Clinical Medicine · 12.11.2026 · 15 (22) · 8866
DOI: 10.3390/jcm15228866 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 12.11.2026. Verfasst von Mikhail Kolev und 4 weiteren Autor:innen. Die Arbeit nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Regional Development Fund.
Abstract (Originalsprache)
Small public multimodal datasets invite overstated claims that wearable signals classify attention-deficit/hyperactivity disorder (ADHD). The HYPERAKTIV dataset served as a methodological case study: 103 adult psychiatric outpatients, 51 with ADHD and 52 clinical controls, with actigraphy, heart-rate recordings, and cognitive testing. The task was differential classification, not screening. Background/Objectives: The objective is to quantify how far the reported performance depends on the evaluation protocol.
Methods
The methods evaluated four nested feature blocks on 63 complete cases with one fixed model and one estimand: the mean out-of-fold area under the receiver operating characteristic curve (AUROC) over 20 fold assignments. Each block was compared with a permutation null from the identical protocol.
Results
The results are that the fold assignment alone moved a single block’s AUROC by up to 0.235, and no block exceeded its null. The heart-rate-variability increment was −0.039 (percentile interval −0.153 to 0.155), inconclusive under every interval method. A bootstrap holding the fitted models fixed suggested a bounded null, and DeLong’s test made the cognitive increment significant; resampling the whole pipeline confirmed neither. On datasets with the heart-rate-variability block permuted, these two tests rejected in 18.3% and 14.7% of those datasets at a 5% level. The pre-scored ADHD Confidence Index alone reached an AUROC of 0.743, descriptively above every fitted model.
Conclusions
The conclusions drawn are that small-sample studies should report a protocol-matched null and intervals that refit the whole pipeline.
Förderung
- European Regional Development Fund
Zitieren
APA
Kolev, M., Naskinova, I., Lazarova, M., Kalinov, H., & Klisurov, V. (2026). Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders. Journal of Clinical Medicine, 15(22), 8866. https://doi.org/10.3390/jcm15228866
Vancouver
Kolev M, Naskinova I, Lazarova M, Kalinov H, Klisurov V. Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders. Journal of Clinical Medicine. 2026;15(22):8866. doi:10.3390/jcm15228866
BibTeX
@article{Kolev2026,
author = {Kolev, Mikhail and Naskinova, Irina and Lazarova, Meglena and Kalinov, Hristo and Klisurov, Velin},
title = {Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders},
journal = {Journal of Clinical Medicine},
year = {2026},
volume = {15},
number = {22},
pages = {8866},
doi = {10.3390/jcm15228866}
}
EUVIMED