Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person Generalization
Methods of Information in Medicine · 01.09.2025 · 64 (03/06) · 129-133
DOI: 10.1055/a-2895-5222 ↗Kurz gefasst
Fachartikel in der Zeitschrift Methods of Information in Medicine, erschienen am 01.09.2025. Verfasst von Minoru Hattori und 1 weiteren Autor:innen. Beteiligte Einrichtung: Graduate School of Biomedical and Health Sciences, Hiroshima University, Department of Medical Education, Japan, Hiroshima. Hauptthemen laut MEDLINE: Phenotype.
Abstract (Originalsprache)
Objectives
A publicly shared digital phenotyping dataset was recently used to train models predicting day-before-panic (DBP) events, reporting a cross-validated receiver operating characteristic area under the curve (ROC-AUC) of around 0.90. Because the stratified five-fold cross-validation (CV) allowed the same participant's days in both training and test folds, it was unclear whether this reflects between-person generalization or information leakage. We reanalyzed the public dataset to quantify this distinction.
Methods
We reanalyzed the public dataset (3,969 person-days; 254 DBP events, 6.4%, across 19 of 43 participants) with the same XGBoost classifier and 71 features, comparing stratified five-fold cross-validation with leave-one-subject-out (LOSO) cross-validation. A retrospective diagnostic decomposition was applied within-subject z-scoring to dynamic features before LOSO.
Results
We reproduced the original ROC-AUC at 0.895 ± 0.011. Under LOSO, pooled ROC-AUC dropped to 0.489 (bootstrap 95% CI: 0.325–0.643, including 0.50). The SHapley Additive exPlanations (SHAP) ranking under stratified CV was dominated by static traits (CTQ subscales, BRIAN, SPAQ). The diagnostic decomposition yielded LOSO ROC-AUC 0.516 (precision–recall area under the curve [PR-AUC] 0.114), suggesting a weak residual within-person signal, though this should not be interpreted as prospective performance.
Conclusion
Robust between-person generalization was not confirmed under subject-held-out evaluation. The feature ranking was trait-dominated, though the data cannot definitively distinguish identity discrimination from other between-person variance sources. Digital phenotyping studies should report subject-level cross-validation alongside standard metrics, per the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) and Prediction model Risk of Bias Assessment Tool (PROBAST) guidelines.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Graduate School of Biomedical and Health Sciences, Hiroshima University, Department of Medical Education, Japan, Hiroshima.
ORCID-Profile: Minoru Hattori
Zitieren
APA
Hattori, M., & Hasunuma, N. (2025). Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person Generalization. Methods of Information in Medicine, 64(03/06), 129-133. https://doi.org/10.1055/a-2895-5222
Vancouver
Hattori M, Hasunuma N. Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person Generalization. Methods of Information in Medicine. 2025;64(03/06):129-133. doi:10.1055/a-2895-5222
BibTeX
@article{Hattori2025,
author = {Hattori, Minoru and Hasunuma, Naoko},
title = {Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person Generalization},
journal = {Methods of Information in Medicine},
year = {2025},
volume = {64},
number = {03/06},
pages = {129--133},
doi = {10.1055/a-2895-5222}
}
EUVIMED