Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering
Frontiers in Digital Health · 04.11.2026 · 8
DOI: 10.3389/fdgth.2026.1956469 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 04.11.2026. Verfasst von Adnan Agha und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Comparisons between general-purpose and medically specialised language models are usually made between fixed checkpoints at a single operating point, varying model scale, training data, and inference scaffolding simultaneously, while leaving the fine-tuning configuration uncontrolled.
Methods
Starting from one 27-billion-parameter parent checkpoint and holding the scale, architecture, evaluation items, decoding, and scoring fixed, we evaluated five fine-tuned derivatives differing in reasoning-trace content, trace length, and training weight. All were evaluated on the identical 2,450-item MedXpertQA-Text test set, providing paired item-level comparisons tested by McNemar exact tests.
Results
The MedXpertQA-Text test accuracy spanned 16.86 percentage points across the parent and its derivatives (43.76%–26.90%) and 15.30 points among the derivatives themselves. Because configurations differ in several training dimensions and one run was performed for each, these are checkpoint-level differences and do not isolate any single training variable. A range of this magnitude can complicate comparisons between unmatched generalist and specialist checkpoints. No derivative exceeded the parent's accuracy. Aggregate accuracy proved a poor summary of change: Four derivatives finishing within 2.41 points of the parent differed from it on 15.1% to 18.2% of items, and on 12.8% to 14.5% after excluding items on which either model reached the generation cap. Retention rates ranged from 47.2% to 80.3%.
Conclusion
No consistent descriptive relationship between rationale length and accuracy was observed, and no category-level retention differences survived the multiplicity correction. In conclusion, aggregate accuracy reports the net direction of change after fine-tuning, but not its magnitude, and can conceal substantial redistribution in which questions a model answers correctly. When a parent model and its fine-tuned derivatives are evaluated on identical items, retention, acquisition, and correctness turnover are directly computable and should be reported along with the aggregate accuracy.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates.
Zitieren
APA
Agha, A., & Anwar, E. (2026). Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering. Frontiers in Digital Health, 8. https://doi.org/10.3389/fdgth.2026.1956469
Vancouver
Agha A, Anwar E. Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering. Frontiers in Digital Health. 2026;8. doi:10.3389/fdgth.2026.1956469
BibTeX
@article{Agha2026,
author = {Agha, Adnan and Anwar, Eram},
title = {Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering},
journal = {Frontiers in Digital Health},
year = {2026},
volume = {8},
doi = {10.3389/fdgth.2026.1956469}
}
EUVIMED