EUVIMEDOpdag EUVIMED
EvaluationsstudieOpen access

Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study

Clara Frydman-Gani iD, Alejandro Arias iD, Maria Perez-Vallejo iD, John Daniel Londoño Martínez iD, Johanna Valencia-Echeverry iD, Mauricio Castano Ramírez iD, Alex A T Bui iD, Nelson B Freimer iD, Carlos Lopez-Jaramillo iD, Loes M Olde Loohuis iD

Journal of Medical Internet Research · 2026

DOI: 10.2196/93328 ↗

At a glance

Evaluationsstudie in Journal of Medical Internet Research, 2026. Written by Clara Frydman-Gani and 9 co-authors. Institution: Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, …. Major MEDLINE topics: Large Language Models, Electronic Health Records, Natural Language Processing, Mental Disorders. 52 references. The full text is freely available in PubMed Central. Funded by: NIMH NIH HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging.

Objective

We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information.

Methods

We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia.

Results

We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro-F1; few-shot: 0.2-0.74 macro-F1), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F1=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F1 decrease of 0.02-0.16).

Conclusions

This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsElectronic Health RecordsNatural Language ProcessingMental DisordersLanguageHumansColombia

Author keywords

EHRLLMsNLPSpanishclinical informaticsconcept detectionconcept extractionelectronic health recordgenerative AIlarge language modelsnatural language processingpsychiatrysynthetic data

Authors and institutions

  • Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States.
  • Research Group in Psychiatry (GIPSI), Department of Psychiatry, School of Medicine, Universidad de Antioquia, Medellin, Antioquia, Colombia.
  • Department of Mental Health and Human Behavior, University of Caldas, Manizales, Caldas, Colombia.
  • Department of Radiological Sciences, University of California, Los Angeles, Los Angeles, CA, United States.

ORCID profiles: Clara Frydman-Gani, Alejandro Arias, Maria Perez-Vallejo, John Daniel Londoño Martínez, Johanna Valencia-Echeverry, Mauricio Castano Ramírez, Alex A T Bui, Nelson B Freimer, Carlos Lopez-Jaramillo, Loes M Olde Loohuis

Funding

  • NIMH NIH HHS

Cite

APA

Frydman-Gani, C., Arias, A., Perez-Vallejo, M., Londoño Martínez, J. D., Valencia-Echeverry, J., Castano Ramírez, M., Bui, A. A. T., Freimer, N. B., Lopez-Jaramillo, C., & Olde Loohuis, L. M. (2026). Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/93328

Vancouver

Frydman-Gani C, Arias A, Perez-Vallejo M, Londoño Martínez JD, Valencia-Echeverry J, Castano Ramírez M, et al. Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. 2026. doi:10.2196/93328

BibTeX
@article{FrydmanGani2026,
  author = {Frydman-Gani, Clara and Arias, Alejandro and Perez-Vallejo, Maria and Londoño Martínez, John Daniel and Valencia-Echeverry, Johanna and Castano Ramírez, Mauricio and Bui, Alex A T and Freimer, Nelson B and Lopez-Jaramillo, Carlos and Olde Loohuis, Loes M},
  title = {Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/93328}
}
Start for free →