EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders

Mikhail Kolev, Irina Naskinova, Meglena Lazarova, Hristo Kalinov, Velin Klisurov

Journal of Clinical Medicine · 12.11.2026 · 15 (22) · 8866

DOI: 10.3390/jcm15228866 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 12.11.2026. Verfasst von Mikhail Kolev und 4 weiteren Autor:innen. Die Arbeit nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Regional Development Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Small public multimodal datasets invite overstated claims that wearable signals classify attention-deficit/hyperactivity disorder (ADHD). The HYPERAKTIV dataset served as a methodological case study: 103 adult psychiatric outpatients, 51 with ADHD and 52 clinical controls, with actigraphy, heart-rate recordings, and cognitive testing. The task was differential classification, not screening. Background/Objectives: The objective is to quantify how far the reported performance depends on the evaluation protocol.

Methods

The methods evaluated four nested feature blocks on 63 complete cases with one fixed model and one estimand: the mean out-of-fold area under the receiver operating characteristic curve (AUROC) over 20 fold assignments. Each block was compared with a permutation null from the identical protocol.

Results

The results are that the fold assignment alone moved a single block’s AUROC by up to 0.235, and no block exceeded its null. The heart-rate-variability increment was −0.039 (percentile interval −0.153 to 0.155), inconclusive under every interval method. A bootstrap holding the fitted models fixed suggested a bounded null, and DeLong’s test made the cognitive increment significant; resampling the whole pipeline confirmed neither. On datasets with the heart-rate-variability block permuted, these two tests rejected in 18.3% and 14.7% of those datasets at a 5% level. The pre-scored ADHD Confidence Index alone reached an AUROC of 0.743, descriptively above every fitted model.

Conclusions

The conclusions drawn are that small-sample studies should report a protocol-matched null and intervals that refit the whole pipeline.

Förderung

  • European Regional Development Fund

Zitieren

APA

Kolev, M., Naskinova, I., Lazarova, M., Kalinov, H., & Klisurov, V. (2026). Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders. Journal of Clinical Medicine, 15(22), 8866. https://doi.org/10.3390/jcm15228866

Vancouver

Kolev M, Naskinova I, Lazarova M, Kalinov H, Klisurov V. Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders. Journal of Clinical Medicine. 2026;15(22):8866. doi:10.3390/jcm15228866

BibTeX
@article{Kolev2026,
  author = {Kolev, Mikhail and Naskinova, Irina and Lazarova, Meglena and Kalinov, Hristo and Klisurov, Velin},
  title = {Instability and Overinterpretation of Predictive Performance in a Small Multimodal Clinical Dataset: A Methodological Case Study of Adult ADHD Versus Other Psychiatric Disorders},
  journal = {Journal of Clinical Medicine},
  year = {2026},
  volume = {15},
  number = {22},
  pages = {8866},
  doi = {10.3390/jcm15228866}
}
Kostenlos starten →