LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation
Journal of Computer-Assisted Linguistic Research · 18.11.2026 · 10 · 1-28
DOI: 10.4995/jclr.2026.26224 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computer-Assisted Linguistic Research, online erschienen am 18.11.2026. Verfasst von Ovidia Martínez Sánchez und 1 weiteren Autor:innen. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Generalitat Valenciana, Agencia Estatal de Investigación.
Abstract (Originalsprache)
Named entity annotation in clinical text is a costly and time-intensive task, particularly for low-resource languages such as Spanish. This article introduces CLINFEM, an XML-annotated corpus comprising 1,200 Spanish medical reports from two regions of Spain. CLINFEM was created using a thorough human-in-the-loop annotation process combining large language model pre-annotation and systematic expert review. It is imperative to acknowledge that each of the 1,200 medical reports underwent meticulous review and correction by a trained human expert. Large language model pre-annotation was employed solely for the purpose of saving time and was never intended to serve as a substitute for manual annotation. Consequently, the entire CLINFEM corpus is a human-validated, gold-standard resource. GPT-4o and Gemini 1.5 Pro were evaluated as pre-annotators using a custom scheme comprising eight clinical entity types and one relational category. GPT-4o achieved a Cohen's κ of 0.73 against human annotation, while Gemini 1.5 Pro obtained κ = 0.41. Human expert review yielded an overall inter-annotator agreement of κ = 0.869 (F1 = 0.793), with a gold-standard subset agreement of κ = 0.9894. This approach reduced the overall annotation effort by around 74% compared to a fully manual approach (81% for the bulk phase alone), saving over 1,000 hours. Error rates varied by medical speciality, with oncology requiring the highest correction rate (94.3% of reports). These results demonstrate that using large language models to assist annotation is a viable and efficient strategy for constructing clinical corpora in Spanish, while also highlighting the continued necessity of human oversight.
Förderung
- Generalitat Valenciana
- Agencia Estatal de Investigación
Zitieren
APA
Sánchez, O. M., & Colorado, B. N. (2026). LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation. Journal of Computer-Assisted Linguistic Research, 10, 1-28. https://doi.org/10.4995/jclr.2026.26224
Vancouver
Sánchez OM, Colorado BN. LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation. Journal of Computer-Assisted Linguistic Research. 2026;10:1-28. doi:10.4995/jclr.2026.26224
BibTeX
@article{Snchez2026,
author = {Sánchez, Ovidia Martínez and Colorado, Borja Navarro},
title = {LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation},
journal = {Journal of Computer-Assisted Linguistic Research},
year = {2026},
volume = {10},
pages = {1--28},
doi = {10.4995/jclr.2026.26224}
}
EUVIMED