A Novel Cognitive Behavioral Therapy–Based Generative AI Tool (Socrates 2.0) to Facilitate Socratic Dialogue: Protocol for a Mixed Methods Feasibility Study
JMIR Research Protocols · 2024
DOI: 10.2196/58195 ↗At a glance
Mixed-Methods-Studie in JMIR Research Protocols, 2024. Written by Philip Held and 5 co-authors. Institution: Department of Psychiatry and Behavioral Sciences, Rush University Medical Center, Chicago, IL, United States. Major MEDLINE topics: Feasibility Studies, Artificial Intelligence, Cognitive Behavioral Therapy. Cited 29 times so far (Crossref); 35 references. The full text is freely available in PubMed Central. Funded by: AHRQ HHS.
Abstract (original language)
Background
Digital mental health tools, designed to augment traditional mental health treatments, are becoming increasingly important due to a wide range of barriers to accessing mental health care, including a growing shortage of clinicians. Most existing tools use rule-based algorithms, often leading to interactions that feel unnatural compared with human therapists. Large language models (LLMs) offer a solution for the development of more natural, engaging digital tools. In this paper, we detail the development of Socrates 2.0, which was designed to engage users in Socratic dialogue surrounding unrealistic or unhelpful beliefs, a core technique in cognitive behavioral therapies. The multiagent LLM-based tool features an artificial intelligence (AI) therapist, Socrates, which receives automated feedback from an AI supervisor and an AI rater. The combination of multiple agents appeared to help address common LLM issues such as looping, and it improved the overall dialogue experience. Initial user feedback from individuals with lived experiences of mental health problems as well as cognitive behavioral therapists has been positive. Moreover, tests in approximately 500 scenarios showed that Socrates 2.0 engaged in harmful responses in under 1% of cases, with the AI supervisor promptly correcting the dialogue each time. However, formal feasibility studies with potential end users are needed.
Objective
This mixed methods study examines the feasibility of Socrates 2.0.
Methods
On the basis of the initial data, we devised a formal feasibility study of Socrates 2.0 to gather qualitative and quantitative data about users’ and clinicians’ experience of interacting with the tool. Using a mixed method approach, the goal is to gather feasibility and acceptability data from 100 users and 50 clinicians to inform the eventual implementation of generative AI tools, such as Socrates 2.0, in mental health treatment. We designed this study to better understand how users and clinicians interact with the tool, including the frequency, length, and time of interactions, users’ satisfaction with the tool overall, quality of each dialogue and individual responses, as well as ways in which the tool should be improved before it is used in efficacy trials. Descriptive and inferential analyses will be performed on data from validated usability measures. Thematic analysis will be performed on the qualitative data.
Results
Recruitment will begin in February 2024 and is expected to conclude by February 2025. As of September 25, 2024, overall, 55 participants have been recruited.
Conclusions
The development of Socrates 2.0 and the outlined feasibility study are important first steps in applying generative AI to mental health treatment delivery and lay the foundation for formal feasibility studies. International Registered Report Identifier (IRRID) DERR1-10.2196/58195
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Psychiatry and Behavioral Sciences, Rush University Medical Center, Chicago, IL, United States.
ORCID profiles: Philip Held, Sarah A Pridgen, Yaozhong Chen, Zuhaib Akhtar, Darpan Amin, Sean Pohorence
Funding
- AHRQ HHS
Cite
APA
Held, P., Pridgen, S. A., Chen, Y., Akhtar, Z., Amin, D., & Pohorence, S. (2024). A Novel Cognitive Behavioral Therapy–Based Generative AI Tool (Socrates 2.0) to Facilitate Socratic Dialogue: Protocol for a Mixed Methods Feasibility Study. JMIR Research Protocols. https://doi.org/10.2196/58195
Vancouver
Held P, Pridgen SA, Chen Y, Akhtar Z, Amin D, Pohorence S. A Novel Cognitive Behavioral Therapy–Based Generative AI Tool (Socrates 2.0) to Facilitate Socratic Dialogue: Protocol for a Mixed Methods Feasibility Study. JMIR Research Protocols. 2024. doi:10.2196/58195
BibTeX
@article{Held2024,
author = {Held, Philip and Pridgen, Sarah A and Chen, Yaozhong and Akhtar, Zuhaib and Amin, Darpan and Pohorence, Sean},
title = {A Novel Cognitive Behavioral Therapy–Based Generative AI Tool (Socrates 2.0) to Facilitate Socratic Dialogue: Protocol for a Mixed Methods Feasibility Study},
journal = {JMIR Research Protocols},
year = {2024},
doi = {10.2196/58195}
}
EUVIMED