EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering

Adnan Agha, Eram Anwar

Frontiers in Digital Health · 04.11.2026 · 8

DOI: 10.3389/fdgth.2026.1956469 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 04.11.2026. Verfasst von Adnan Agha und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Comparisons between general-purpose and medically specialised language models are usually made between fixed checkpoints at a single operating point, varying model scale, training data, and inference scaffolding simultaneously, while leaving the fine-tuning configuration uncontrolled.

Methods

Starting from one 27-billion-parameter parent checkpoint and holding the scale, architecture, evaluation items, decoding, and scoring fixed, we evaluated five fine-tuned derivatives differing in reasoning-trace content, trace length, and training weight. All were evaluated on the identical 2,450-item MedXpertQA-Text test set, providing paired item-level comparisons tested by McNemar exact tests.

Results

The MedXpertQA-Text test accuracy spanned 16.86 percentage points across the parent and its derivatives (43.76%–26.90%) and 15.30 points among the derivatives themselves. Because configurations differ in several training dimensions and one run was performed for each, these are checkpoint-level differences and do not isolate any single training variable. A range of this magnitude can complicate comparisons between unmatched generalist and specialist checkpoints. No derivative exceeded the parent's accuracy. Aggregate accuracy proved a poor summary of change: Four derivatives finishing within 2.41 points of the parent differed from it on 15.1% to 18.2% of items, and on 12.8% to 14.5% after excluding items on which either model reached the generation cap. Retention rates ranged from 47.2% to 80.3%.

Conclusion

No consistent descriptive relationship between rationale length and accuracy was observed, and no category-level retention differences survived the multiplicity correction. In conclusion, aggregate accuracy reports the net direction of change after fine-tuning, but not its magnitude, and can conceal substantial redistribution in which questions a model answers correctly. When a parent model and its fine-tuned derivatives are evaluated on identical items, retention, acquisition, and correctness turnover are directly computable and should be reported along with the aggregate accuracy.

Schlagwörter

Stichwörter der Autor:innen

benchmarkingclinical natural language processingfine-tuninglarge language modelsmachine learningmedical question-answeringmodel evaluation

Autor:innen und Institutionen

  • Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates.

Zitieren

APA

Agha, A., & Anwar, E. (2026). Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering. Frontiers in Digital Health, 8. https://doi.org/10.3389/fdgth.2026.1956469

Vancouver

Agha A, Anwar E. Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering. Frontiers in Digital Health. 2026;8. doi:10.3389/fdgth.2026.1956469

BibTeX
@article{Agha2026,
  author = {Agha, Adnan and Anwar, Eram},
  title = {Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering},
  journal = {Frontiers in Digital Health},
  year = {2026},
  volume = {8},
  doi = {10.3389/fdgth.2026.1956469}
}
Kostenlos starten →