EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering

Sebastian Daniel Boie iD, Florian Reis, Nicolas Frey, Elias Grünewald, Felix Balzer

Journal of Medical Systems · 26.06.2026 · 50 (1)

DOI: 10.1007/s10916-026-02430-0 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 26.06.2026. Verfasst von Sebastian Daniel Boie und 4 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität z…. Hauptthemen laut MEDLINE: Large Language Models, Self Report. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Charité - Universitätsmedizin Berlin.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions. Materials and Methods: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence.

Results

Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties (“Skin” best: 0.041; “Social & Preventive Medicine” worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties.

Discussion

Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains.

Conclusion

We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved “self-knowledge”, but uneven overconfidence persists.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsSelf ReportHumansCalibration

Stichwörter der Autor:innen

AccuracyCalibrationConfidenceLarge language model

Autor:innen und Institutionen

  • Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany. sebastian-daniel.boi…
  • Department of Anesthesiology and Intensive Care Medicine (CCM/CVK), Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany. sebastian…
  • Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany.

ORCID-Profile: Sebastian Daniel Boie

Förderung

  • Charité - Universitätsmedizin Berlin

Zitieren

APA

Boie, S. D., Reis, F., Frey, N., Grünewald, E., & Balzer, F. (2026). Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02430-0

Vancouver

Boie SD, Reis F, Frey N, Grünewald E, Balzer F. Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02430-0

BibTeX
@article{Boie2026,
  author = {Boie, Sebastian Daniel and Reis, Florian and Frey, Nicolas and Grünewald, Elias and Balzer, Felix},
  title = {Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering},
  journal = {Journal of Medical Systems},
  year = {2026},
  volume = {50},
  number = {1},
  doi = {10.1007/s10916-026-02430-0}
}
Kostenlos starten →