Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint)
JMIR Formative Research · 2025
DOI: 10.2196/87054 ↗Kurz gefasst
Beobachtungsstudie in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Matthew So und 2 weiteren Autor:innen. Beteiligte Einrichtung: Okaya, 540 Mohawk Drive, Boulder, CO, 80303, United States, 1 9259841028. Hauptthemen laut MEDLINE: Fatigue, Biomarkers, Mental Health, Cognition. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Collection of multimodal data (video, audio, and text) can yield digital biomarkers relevant to mental health, fatigue, and cognition. However, the feasibility and signal characteristics in operational populations remain underexplored.
Objective
The objectives of this study were to (1) extract an evidence-based library of vision, speech, and language features; (2) assess the feasibility of a fully remote conversational platform (Okaya) for collecting analyzable multimodal data; and (3) conduct preliminary signal checks for depression, fatigue, and cognition.
Methods
Participants were recruited from the US Air Force and US Space Force. All participants completed the Okaya check-in, which included a voice conversation with a large language model. A total of 66 visual, acoustic, and text features were extracted from each interaction between the participant and the large language model. For validation purposes, the study also collected measures of depression (Patient Health Questionnaire-9), fatigue (Cancer Fatigue Scale), and cognition (trail making test). We evaluated the feasibility of the platform and correlation between the extracted features and the validated assessments.
Results
A total of 8 unique participants contributed with 62 sessions over a period from March 6, 2025, to August 6, 2025. The platform was deemed feasible as 6 of the 8 participants opted to complete more than one session, and the 3 participants who provided feedback reported high overall experience and usability. From the data perspective, preliminary correlations produced significant results for multiple potential digital biomarkers, including (1) pitch (P=.047), volume SD (P=.04), volume slope (P=.04), automated readability index complexity (P=.047), Flesch-Kincaid complexity (P=.04), and Gunning Fog complexity (P=.04) for depression; (2) pitch (P=.009), volume SD (P=.007), volume slope (P=.02), average F2 formant frequency (P=.03), Gunning Fog complexity (P=.049), and eyelid droop (P=.047) for fatigue; and (3) shimmer (P=.03) for cognition. We also observed how features varied over time among participants with multiple sessions.
Conclusions
The conversational and artificial intelligence-enabled platform was feasible among an operational sample as a method to collect multimodal data correlated with depression, fatigue, and cognition. These results align with those for previously discovered digital biomarkers of mental health, fatigue, and cognition and inform the development of personalized models for each user while detecting anomalies in a remote monitoring setting.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Substanzen
Autor:innen und Institutionen
- Okaya, 540 Mohawk Drive, Boulder, CO, 80303, United States, 1 9259841028.
- Cornell Tech, New York, NY, United States.
ORCID-Profile: Matthew So, Michael Sobolev, Gregory Menvielle
Zitieren
APA
So, M., Sobolev, M., & Menvielle, G. (2025). Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint). JMIR Formative Research. https://doi.org/10.2196/87054
Vancouver
So M, Sobolev M, Menvielle G. Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint). JMIR Formative Research. 2025. doi:10.2196/87054
BibTeX
@article{So2025,
author = {So, Matthew and Sobolev, Michael and Menvielle, Gregory},
title = {Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint)},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/87054}
}
EUVIMED