EUVIMEDEUVIMED entdecken
BeobachtungsstudieOpen Access

Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint)

Matthew So iD, Michael Sobolev iD, Gregory Menvielle iD

JMIR Formative Research · 2025

DOI: 10.2196/87054 ↗

Kurz gefasst

Beobachtungsstudie in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Matthew So und 2 weiteren Autor:innen. Beteiligte Einrichtung: Okaya, 540 Mohawk Drive, Boulder, CO, 80303, United States, 1 9259841028. Hauptthemen laut MEDLINE: Fatigue, Biomarkers, Mental Health, Cognition. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Collection of multimodal data (video, audio, and text) can yield digital biomarkers relevant to mental health, fatigue, and cognition. However, the feasibility and signal characteristics in operational populations remain underexplored.

Objective

The objectives of this study were to (1) extract an evidence-based library of vision, speech, and language features; (2) assess the feasibility of a fully remote conversational platform (Okaya) for collecting analyzable multimodal data; and (3) conduct preliminary signal checks for depression, fatigue, and cognition.

Methods

Participants were recruited from the US Air Force and US Space Force. All participants completed the Okaya check-in, which included a voice conversation with a large language model. A total of 66 visual, acoustic, and text features were extracted from each interaction between the participant and the large language model. For validation purposes, the study also collected measures of depression (Patient Health Questionnaire-9), fatigue (Cancer Fatigue Scale), and cognition (trail making test). We evaluated the feasibility of the platform and correlation between the extracted features and the validated assessments.

Results

A total of 8 unique participants contributed with 62 sessions over a period from March 6, 2025, to August 6, 2025. The platform was deemed feasible as 6 of the 8 participants opted to complete more than one session, and the 3 participants who provided feedback reported high overall experience and usability. From the data perspective, preliminary correlations produced significant results for multiple potential digital biomarkers, including (1) pitch (P=.047), volume SD (P=.04), volume slope (P=.04), automated readability index complexity (P=.047), Flesch-Kincaid complexity (P=.04), and Gunning Fog complexity (P=.04) for depression; (2) pitch (P=.009), volume SD (P=.007), volume slope (P=.02), average F2 formant frequency (P=.03), Gunning Fog complexity (P=.049), and eyelid droop (P=.047) for fatigue; and (3) shimmer (P=.03) for cognition. We also observed how features varied over time among participants with multiple sessions.

Conclusions

The conversational and artificial intelligence-enabled platform was feasible among an operational sample as a method to collect multimodal data correlated with depression, fatigue, and cognition. These results align with those for previously discovered digital biomarkers of mental health, fatigue, and cognition and inform the development of personalized models for each user while detecting anomalies in a remote monitoring setting.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

FatigueBiomarkersMental HealthCognitionHumansFeasibility StudiesMaleFemaleAdultMiddle AgedDigital HealthDepression

Stichwörter der Autor:innen

AIartificial intelligencecomputer visiondigital biomarkersmultimodal sensingnatural language processingspeech acoustics

Substanzen

Biomarkers

Autor:innen und Institutionen

  • Okaya, 540 Mohawk Drive, Boulder, CO, 80303, United States, 1 9259841028.
  • Cornell Tech, New York, NY, United States.

ORCID-Profile: Matthew So, Michael Sobolev, Gregory Menvielle

Zitieren

APA

So, M., Sobolev, M., & Menvielle, G. (2025). Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint). JMIR Formative Research. https://doi.org/10.2196/87054

Vancouver

So M, Sobolev M, Menvielle G. Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint). JMIR Formative Research. 2025. doi:10.2196/87054

BibTeX
@article{So2025,
  author = {So, Matthew and Sobolev, Michael and Menvielle, Gregory},
  title = {Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study (Preprint)},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/87054}
}
Kostenlos starten →