Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering
Journal of Medical Systems · 26.06.2026 · 50 (1)
DOI: 10.1007/s10916-026-02430-0 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 26.06.2026. Verfasst von Sebastian Daniel Boie und 4 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität z…. Hauptthemen laut MEDLINE: Large Language Models, Self Report. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Charité - Universitätsmedizin Berlin.
Abstract (Originalsprache)
Objective
To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions. Materials and Methods: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence.
Results
Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties (“Skin” best: 0.041; “Social & Preventive Medicine” worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties.
Discussion
Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains.
Conclusion
We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved “self-knowledge”, but uneven overconfidence persists.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany. sebastian-daniel.boi…
- Department of Anesthesiology and Intensive Care Medicine (CCM/CVK), Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany. sebastian…
- Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany.
ORCID-Profile: Sebastian Daniel Boie
Förderung
- Charité - Universitätsmedizin Berlin
Zitieren
APA
Boie, S. D., Reis, F., Frey, N., Grünewald, E., & Balzer, F. (2026). Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02430-0
Vancouver
Boie SD, Reis F, Frey N, Grünewald E, Balzer F. Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02430-0
BibTeX
@article{Boie2026,
author = {Boie, Sebastian Daniel and Reis, Florian and Frey, Nicolas and Grünewald, Elias and Balzer, Felix},
title = {Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering},
journal = {Journal of Medical Systems},
year = {2026},
volume = {50},
number = {1},
doi = {10.1007/s10916-026-02430-0}
}
EUVIMED