Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review
JMIR Research Protocols · 2026-09-14 · 15 · e91677-e91677
DOI: 10.2196/91677 ↗At a glance
Scoping Review in JMIR Research Protocols, published online on 2026-09-14. Written by Antonio Salinas-Layana and 8 co-authors. Institution: Doctorado en Psicoterapia, Universidad de Chile y Pontificia Universidad Católica de Chile, Santiago, Región Metropolitana, Chile. Major MEDLINE topics: Large Language Models. 26 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Digital mental health interventions (DMHIs) can help close persistent gaps in access to assessment, prevention, and treatment. Recent advances in generative AI, particularly large language models (LLMs), further expand this promise by enabling natural language understanding, personalization, and empathic interaction across assessment, support, and therapeutic contexts. However, significant evaluation challenges persist, including a lack of standardized constructs and validated instruments, which limit the comparability, reproducibility, and generalizability of the findings. No systematic synthesis currently documents the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs, thereby hampering the development of a comprehensive evidence base to guide future evaluation efforts.
Objective
This scoping review aims to systematically map and synthesize the available evidence on frameworks, methodologies, and tools used to evaluate LLMs applied to DMHIs. Specifically, it aims to identify the constructs assessed, the instruments used, and the evaluation procedures and stages addressed.
Methods
Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, this scoping review will search 5 electronic databases (PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library) from January 1, 2019, to September 15, 2025. Eligibility criteria will encompass both empirical studies and theoretical proposals evaluating LLMs embedded within DMHIs. Studies limited to risk detection or decision support systems without an intervention component will be excluded. Data extraction will capture information on conceptual frameworks, methodological designs, evaluation procedures and tools, measured constructs, and other relevant contextual information.
Results
The systematic search was conducted between September 1 and 15, 2025, yielding 4273 records across the 5 databases. After duplicate removal, of the 4273 records, 2980 (69.7%) remained for screening. A pilot screening exercise involving 4 independent reviewers achieved high interrater reliability (free-marginal Randolph κ=0.81), with 76% (19/25 of the pilot sample) unanimous agreement, indicating adequate calibration of selection criteria. These figures are interim process indicators rather than final review findings as title and abstract screening of the remaining records is currently underway.
Conclusions
This scoping review is expected to provide one of the first systematic syntheses of frameworks, methodologies, and tools used to evaluate LLMs in DMHIs. By identifying prevailing patterns and gaps, the resulting evidence map is intended to serve as a practical reference for researchers, developers, and policymakers working toward a scientifically grounded, safe, ethical, and effective deployment of LLMs in mental health interventions.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Doctorado en Psicoterapia, Universidad de Chile y Pontificia Universidad Católica de Chile, Santiago, Región Metropolitana, Chile.
- Nucleus to Improve the Mental Health of Adolescents and Youths (Imhay), Santiago, Región Metropolitana, Chile.
- Facultad de Psicología y Humanidades, Universidad San Sebastián, Santiago, Región Metropolitana, Chile.
- Center for Research and Action on Social Determination and Mental Health (CIADES), Santiago, Región Metropolitana, Chile.
- Departamento de Ciencias de la Educación, Área de Psicología Social, Universidad de Burgos, Burgos, Castilla y León, Spain.
- VeryMind, Santiago, Región Metropolitana, Chile.
ORCID profiles: Antonio Salinas-Layana, Álvaro Jiménez-Molina, Daniela Lira, Félix Alberto Véliz-Montoya, Alexi Venegas, Nicolás Muñoz, Mario Chandía, Rigoberto Rojas, Vania Martínez
Cite
APA
Salinas-Layana, A., Jiménez-Molina, Á., Lira, D., Véliz-Montoya, F. A., Venegas, A., Muñoz, N., Chandía, M., Rojas, R., & Martínez, V. (2026). Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review. JMIR Research Protocols, 15, e91677-e91677. https://doi.org/10.2196/91677
Vancouver
Salinas-Layana A, Jiménez-Molina Á, Lira D, Véliz-Montoya FA, Venegas A, Muñoz N, et al. Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review. JMIR Research Protocols. 2026;15:e91677-e91677. doi:10.2196/91677
BibTeX
@article{SalinasLayana2026,
author = {Salinas-Layana, Antonio and Jiménez-Molina, Álvaro and Lira, Daniela and Véliz-Montoya, Félix Alberto and Venegas, Alexi and Muñoz, Nicolás and Chandía, Mario and Rojas, Rigoberto and Martínez, Vania},
title = {Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review},
journal = {JMIR Research Protocols},
year = {2026},
volume = {15},
pages = {e91677--e91677},
doi = {10.2196/91677}
}
EUVIMED