EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access

Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study

Clara Frydman-Gani iD, Alejandro Arias iD, Maria Perez-Vallejo iD, John Daniel Londoño Martínez iD, Johanna Valencia-Echeverry iD, Mauricio Castano Ramírez iD, Alex A T Bui iD, Nelson B Freimer iD, Carlos Lopez-Jaramillo iD, Loes M Olde Loohuis iD

Journal of Medical Internet Research · 2026

DOI: 10.2196/93328 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Clara Frydman-Gani und 9 weiteren Autor:innen. Beteiligte Einrichtung: Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, …. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Natural Language Processing, Mental Disorders. Die Arbeit nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging.

Objective

We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information.

Methods

We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia.

Results

We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro-F1; few-shot: 0.2-0.74 macro-F1), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F1=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F1 decrease of 0.02-0.16).

Conclusions

This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsElectronic Health RecordsNatural Language ProcessingMental DisordersLanguageHumansColombia

Stichwörter der Autor:innen

EHRLLMsNLPSpanishclinical informaticsconcept detectionconcept extractionelectronic health recordgenerative AIlarge language modelsnatural language processingpsychiatrysynthetic data

Autor:innen und Institutionen

  • Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States.
  • Research Group in Psychiatry (GIPSI), Department of Psychiatry, School of Medicine, Universidad de Antioquia, Medellin, Antioquia, Colombia.
  • Department of Mental Health and Human Behavior, University of Caldas, Manizales, Caldas, Colombia.
  • Department of Radiological Sciences, University of California, Los Angeles, Los Angeles, CA, United States.

ORCID-Profile: Clara Frydman-Gani, Alejandro Arias, Maria Perez-Vallejo, John Daniel Londoño Martínez, Johanna Valencia-Echeverry, Mauricio Castano Ramírez, Alex A T Bui, Nelson B Freimer, Carlos Lopez-Jaramillo, Loes M Olde Loohuis

Förderung

  • NIMH NIH HHS

Zitieren

APA

Frydman-Gani, C., Arias, A., Perez-Vallejo, M., Londoño Martínez, J. D., Valencia-Echeverry, J., Castano Ramírez, M., Bui, A. A. T., Freimer, N. B., Lopez-Jaramillo, C., & Olde Loohuis, L. M. (2026). Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/93328

Vancouver

Frydman-Gani C, Arias A, Perez-Vallejo M, Londoño Martínez JD, Valencia-Echeverry J, Castano Ramírez M, et al. Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. 2026. doi:10.2196/93328

BibTeX
@article{FrydmanGani2026,
  author = {Frydman-Gani, Clara and Arias, Alejandro and Perez-Vallejo, Maria and Londoño Martínez, John Daniel and Valencia-Echeverry, Johanna and Castano Ramírez, Mauricio and Bui, Alex A T and Freimer, Nelson B and Lopez-Jaramillo, Carlos and Olde Loohuis, Loes M},
  title = {Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/93328}
}
Kostenlos starten →