EUVIMEDOpdag EUVIMED
Open access

Detection of Microbehavior Intervals for Predicting Mental Health: Clinically Relevant and Advanced Multimodal Temporal Analysis

Sapir Gershov iD, Charlotte E Hilberdink iD, Yiwen Zhao iD, Sarah B Birnbaum iD, Victoria Mueller iD, Stephen P Wall iD, Katharina Schultebraucks iD

Journal of Medical Internet Research · 2026-05-27 · 28 · e87049

DOI: 10.2196/87049 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2026-05-27. Written by Sapir Gershov and 6 co-authors. Institution: Department of Psychiatry, Grossman School of Medicine, New York University, New York, NY, United States. Major MEDLINE topics: Stress Disorders, Post-Traumatic, Health Personnel, Mental Health, Burnout, Professional. 65 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Health care workers (HCWs) face sustained psychological demands that place them at heightened risk for burnout and posttraumatic stress disorder (PTSD). However, assessing psychological distress in this population remains challenging because of stigma, underreporting, and the limitations of self-report tools. Although nonverbal behaviors such as facial expressions and gaze hold diagnostic promise, most approaches overlook the fine-grained, temporal fluctuations in these signals. In this study, we focused on microbehavior intervals—brief, involuntary changes in multimodal nonverbal signals—that emerge during emotion-eliciting interviews.

Objective

This study aimed to determine whether microbehavior intervals improve the discrimination of psychological distress profiles among HCWs with symptoms of burnout and PTSD.

Methods

HCWs participated in a semistructured interview that included 5 work-related, emotionally charged questions and that was recorded via Webex (online video platform). Participants also completed validated questionnaires for burnout (Maslach Burnout Inventory General Survey 9-item) and PTSD (PTSD checklist for Diagnostic and Statistical Manual, 5th edition). Recordings were analyzed with computer vision models to generate time-series data of facial expressions, head movement, gaze, body posture, and hand gestures. An unsupervised anomaly detection model (MOMENT [a Family of Open Time-Series Foundation Models]) isolated microbehavior intervals without requiring manual labels. Features derived from these intervals were used to train a deep learning classifier that predicted 4 symptom classes of psychological distress: “moderate-severe burnout,” “subthreshold-provisional PTSD,” “burnout+PTSD,” and “resilient.” We conducted an ablation study by systematically removing one behavioral data stream at a time. Finally, we conducted an explainability analysis to characterize the features driving model predictions.

Results

We analyzed 258 interview recordings from 151 HCWs. Per interview, an average of 19.65 (SD 6.01) microbehavior intervals were detected, each lasting an average of 1.31 (SD 1.10) seconds. The classifier demonstrated robust performance across classes, achieving a macro– F1-score of 0.75 and a macro area under the receiver operating characteristic curve of 0.80 on held-out data. Ablation analysis showed that excluding gaze or arousal-valence signals caused the largest performance declines, particularly in recall and F1-score. The explainability analysis revealed distinct temporal patterns across symptom classes, with irregularity and variability in microbehaviors emerging as key predictors.

Conclusions

Focusing on microbehavior intervals yields a scalable, interpretable, and annotation-free framework for detecting psychological distress from nonverbal signals. By moving from whole-video features to fine-grained multimodal temporal modeling, we successfully captured subtle, involuntary fluctuations in nonverbal responses to emotion-eliciting questions. This multimodal approach enables an objective, robust, and explainable assessment of psychological distress and offers a promising complement to conventional psychometric assessments.

Subjects

MeSH terms (MEDLINE)

Stress Disorders, Post-TraumaticHealth PersonnelMental HealthBurnout, ProfessionalHumansFemaleFacial ExpressionMaleAdultMiddle Aged

Author keywords

PTSDanomaly detectionburnouthealth care workersmultimodalposttraumatic stress disorder

Authors and institutions

  • Department of Psychiatry, Grossman School of Medicine, New York University, New York, NY, United States.
  • Department of Biomedical Engineering, Tandon School of Engineering, New York University, New York, NY, United States.
  • Department of Emergency Medicine, Grossman School of Medicine, New York University, New York, NY, United States.
  • Department of Population Health, Grossman School of Medicine, New York University, New York, NY, United States.

ORCID profiles: Sapir Gershov, Charlotte E Hilberdink, Yiwen Zhao, Sarah B Birnbaum, Victoria Mueller, Stephen P Wall, Katharina Schultebraucks

Cite

APA

Gershov, S., Hilberdink, C. E., Zhao, Y., Birnbaum, S. B., Mueller, V., Wall, S. P., & Schultebraucks, K. (2026). Detection of Microbehavior Intervals for Predicting Mental Health: Clinically Relevant and Advanced Multimodal Temporal Analysis. Journal of Medical Internet Research, 28, e87049. https://doi.org/10.2196/87049

Vancouver

Gershov S, Hilberdink CE, Zhao Y, Birnbaum SB, Mueller V, Wall SP, et al. Detection of Microbehavior Intervals for Predicting Mental Health: Clinically Relevant and Advanced Multimodal Temporal Analysis. Journal of Medical Internet Research. 2026;28:e87049. doi:10.2196/87049

BibTeX
@article{Gershov2026,
  author = {Gershov, Sapir and Hilberdink, Charlotte E and Zhao, Yiwen and Birnbaum, Sarah B and Mueller, Victoria and Wall, Stephen P and Schultebraucks, Katharina},
  title = {Detection of Microbehavior Intervals for Predicting Mental Health: Clinically Relevant and Advanced Multimodal Temporal Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e87049},
  doi = {10.2196/87049}
}
Start for free →