EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study

Clemente García-Hidalgo, Manuel Carpio-Salmerón, Pau Benito, Carlos Carazo-Casas, Beatriz Carratalá, Jaime Campos Pavón

JMIR Medical Education · 29.10.2026 · 12 · e106199-e106199

DOI: 10.2196/106199 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 29.10.2026. Verfasst von Clemente García-Hidalgo und 5 weiteren Autor:innen. Beteiligte Einrichtung: Radiology Department, Hospital General Universitario Morales Meseguer, Av Marqués de los Vélez S/N, Murciaº, Murcia, 30008, Spain, 34 666817007. Hauptthemen laut MEDLINE: Large Language Models, Psychometrics, Benchmarking, Educational Measurement. Die Arbeit nennt 58 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) achieve high scores on health-professions examinations. Whether these scores support the same interpretations as human scores remains uncertain. Spain uses parallel multiple-choice examinations for entry to specialty training in medicine (Médico Interno Residente [MIR]), nursing (Enfermero Interno Residente [EIR]), psychology (Psicólogo Interno Residente [PIR]), and pharmacy (Farmacéutico Interno Residente [FIR]).

Objective

Using the January 2026 MIR, EIR, PIR, and FIR examinations, this study aimed to quantify the correspondence between human and LLM item facility and to describe model accuracy, distractor selection, and performance associated with measured surface features.

Methods

We evaluated 839 valid items using 12 configurations: GPT-5.2 at 3 reasoning efforts and Claude Opus, Sonnet, and Haiku 4.5 (Anthropic), each under neutral and expert prompts, with one run per configuration. The primary measure was the Spearman correlation (ρ) between human and machine facility on 828 complete paired items. We examined sensitivity to panel composition, scoring, pooling, and assumed reliability; compared distractor choices with a post hoc plausibility-weighted baseline; and evaluated a surface-feature classifier. Image analyses distinguished supplied from withheld images. An exploratory LLM-judge audit assessed generated explanations. Bootstrap CIs resample items or, for distractor concordance, error events; they exclude run-to-run uncertainty.

Results

The highest pooled accuracy was 94.8% (795/839); all 12 configurations answered 668 of 828 paired items correctly (80.7%). Scores clustered near the ceiling among the strongest configurations. Human and machine facility correlated at 0.40 (95% CI 0.34-0.45), with estimates of 0.31 to 0.40 across panel definitions and 0.39 (95% CI 0.33-0.45) after exam-stratified pooling. The observed correlation was lower than predicted by a specified one-axis simulation with a uniform model advantage (0.63; predictive interval 0.59 to 0.66; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPsychometricsBenchmarkingEducational MeasurementHumansCross-Sectional StudiesSpainReproducibility of Results

Stichwörter der Autor:innen

AISpainclinical competenceconstruct validityeducational measurementlarge language modelsmedical educationmultiple-choicepsychometricsspecialty-access examination

Autor:innen und Institutionen

  • Radiology Department, Hospital General Universitario Morales Meseguer, Av Marqués de los Vélez S/N, Murciaº, Murcia, 30008, Spain, 34 666817007.
  • Endocrinology Department, Santa Lucía University General Hospital, Cartagena, Murcia, Spain.
  • Department of Preventive Medicine and Epidemiology, Clinical Institute of Medicine and Dermatology (ICMiD), Hospital Clínic de Barcelona, Barcelona, Catalonia, Spain.
  • Barcelona Institute for Global Health (ISGlobal), Universitat de Barcelona, Hospital Clínic de Barcelona, Barcelona, Catalonia, Spain.
  • Department of Otolaryngology, Hospital Universitario Ramón y Cajal, Madrid, Madrid, Spain.
  • Innovation and Digital Projects Academic Department, Healthcademia, Madrid, Madrid, Spain.

Zitieren

APA

García-Hidalgo, C., Carpio-Salmerón, M., Benito, P., Carazo-Casas, C., Carratalá, B., & Campos Pavón, J. (2026). Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study. JMIR Medical Education, 12, e106199-e106199. https://doi.org/10.2196/106199

Vancouver

García-Hidalgo C, Carpio-Salmerón M, Benito P, Carazo-Casas C, Carratalá B, Campos Pavón J. Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study. JMIR Medical Education. 2026;12:e106199-e106199. doi:10.2196/106199

BibTeX
@article{GarcaHidalgo2026,
  author = {García-Hidalgo, Clemente and Carpio-Salmerón, Manuel and Benito, Pau and Carazo-Casas, Carlos and Carratalá, Beatriz and Campos Pavón, Jaime},
  title = {Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e106199--e106199},
  doi = {10.2196/106199}
}
Kostenlos starten →