EUVIMEDOpdag EUVIMED
Scoping ReviewStudienprotokollOpen access

Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review

Antonio Salinas-Layana iD, Álvaro Jiménez-Molina iD, Daniela Lira iD, Félix Alberto Véliz-Montoya iD, Alexi Venegas iD, Nicolás Muñoz iD, Mario Chandía iD, Rigoberto Rojas iD, Vania Martínez iD

JMIR Research Protocols · 2026-09-14 · 15 · e91677-e91677

DOI: 10.2196/91677 ↗

At a glance

Scoping Review in JMIR Research Protocols, published online on 2026-09-14. Written by Antonio Salinas-Layana and 8 co-authors. Institution: Doctorado en Psicoterapia, Universidad de Chile y Pontificia Universidad Católica de Chile, Santiago, Región Metropolitana, Chile. Major MEDLINE topics: Large Language Models. 26 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Digital mental health interventions (DMHIs) can help close persistent gaps in access to assessment, prevention, and treatment. Recent advances in generative AI, particularly large language models (LLMs), further expand this promise by enabling natural language understanding, personalization, and empathic interaction across assessment, support, and therapeutic contexts. However, significant evaluation challenges persist, including a lack of standardized constructs and validated instruments, which limit the comparability, reproducibility, and generalizability of the findings. No systematic synthesis currently documents the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs, thereby hampering the development of a comprehensive evidence base to guide future evaluation efforts.

Objective

This scoping review aims to systematically map and synthesize the available evidence on frameworks, methodologies, and tools used to evaluate LLMs applied to DMHIs. Specifically, it aims to identify the constructs assessed, the instruments used, and the evaluation procedures and stages addressed.

Methods

Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, this scoping review will search 5 electronic databases (PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library) from January 1, 2019, to September 15, 2025. Eligibility criteria will encompass both empirical studies and theoretical proposals evaluating LLMs embedded within DMHIs. Studies limited to risk detection or decision support systems without an intervention component will be excluded. Data extraction will capture information on conceptual frameworks, methodological designs, evaluation procedures and tools, measured constructs, and other relevant contextual information.

Results

The systematic search was conducted between September 1 and 15, 2025, yielding 4273 records across the 5 databases. After duplicate removal, of the 4273 records, 2980 (69.7%) remained for screening. A pilot screening exercise involving 4 independent reviewers achieved high interrater reliability (free-marginal Randolph κ=0.81), with 76% (19/25 of the pilot sample) unanimous agreement, indicating adequate calibration of selection criteria. These figures are interim process indicators rather than final review findings as title and abstract screening of the remaining records is currently underway.

Conclusions

This scoping review is expected to provide one of the first systematic syntheses of frameworks, methodologies, and tools used to evaluate LLMs in DMHIs. By identifying prevailing patterns and gaps, the resulting evidence map is intended to serve as a practical reference for researchers, developers, and policymakers working toward a scientifically grounded, safe, ethical, and effective deployment of LLMs in mental health interventions.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsHumansScoping Reviews as TopicDigital Health

Author keywords

AIartificial intelligencechatbotdigital healthlarge language modelsmental disordersmental healthmental health servicesscoping reviewtelemedicinetherapy, computer assisted

Authors and institutions

  • Doctorado en Psicoterapia, Universidad de Chile y Pontificia Universidad Católica de Chile, Santiago, Región Metropolitana, Chile.
  • Nucleus to Improve the Mental Health of Adolescents and Youths (Imhay), Santiago, Región Metropolitana, Chile.
  • Facultad de Psicología y Humanidades, Universidad San Sebastián, Santiago, Región Metropolitana, Chile.
  • Center for Research and Action on Social Determination and Mental Health (CIADES), Santiago, Región Metropolitana, Chile.
  • Departamento de Ciencias de la Educación, Área de Psicología Social, Universidad de Burgos, Burgos, Castilla y León, Spain.
  • VeryMind, Santiago, Región Metropolitana, Chile.

ORCID profiles: Antonio Salinas-Layana, Álvaro Jiménez-Molina, Daniela Lira, Félix Alberto Véliz-Montoya, Alexi Venegas, Nicolás Muñoz, Mario Chandía, Rigoberto Rojas, Vania Martínez

Cite

APA

Salinas-Layana, A., Jiménez-Molina, Á., Lira, D., Véliz-Montoya, F. A., Venegas, A., Muñoz, N., Chandía, M., Rojas, R., & Martínez, V. (2026). Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review. JMIR Research Protocols, 15, e91677-e91677. https://doi.org/10.2196/91677

Vancouver

Salinas-Layana A, Jiménez-Molina Á, Lira D, Véliz-Montoya FA, Venegas A, Muñoz N, et al. Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review. JMIR Research Protocols. 2026;15:e91677-e91677. doi:10.2196/91677

BibTeX
@article{SalinasLayana2026,
  author = {Salinas-Layana, Antonio and Jiménez-Molina, Álvaro and Lira, Daniela and Véliz-Montoya, Félix Alberto and Venegas, Alexi and Muñoz, Nicolás and Chandía, Mario and Rojas, Rigoberto and Martínez, Vania},
  title = {Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review},
  journal = {JMIR Research Protocols},
  year = {2026},
  volume = {15},
  pages = {e91677--e91677},
  doi = {10.2196/91677}
}
Start for free →