Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers
Journal of Medical Internet Research · 2025
DOI: 10.2196/64452 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Eden Avnat und 14 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622. Hauptthemen laut MEDLINE: Semantics, Benchmarking, Evidence-Based Medicine, Language. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Clinical problem-solving requires processing of semantic medical knowledge, such as illness scripts, and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate nonlanguage evidence-based answers to clinical questions is inherently limited by tokenization.
Objective
This study aimed to evaluate LLMs’ performance on two question types: numeric (correlating findings) and semantic (differentiating entities), while examining differences within and between LLMs in medical aspects and comparing their performance to humans.
Methods
To generate straightforward multichoice questions and answers (Q and As) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (containing data from more than 50,000 peer-reviewed studies) and created the EBM questions and answers (EBMQAs). EBMQA comprises 105,222 Q and As, categorized by medical topics (eg, medical disciplines) and nonmedical topics (eg, question length), and classified into numerical or semantic types. We benchmarked a dataset of 24,000 Q and As on two state-of-the-art LLMs, GPT-4 (OpenAI) and Claude 3 Opus (Anthropic). We evaluated the LLM’s accuracy on semantic and numerical question types and according to sublabeled topics. In addition, we examined the question-answering rate of LLMs by enabling them to choose to abstain from responding to questions. For validation, we compared the results for 100 unrelated numerical EBMQA questions between six human medical experts and the two language models.
Results
In an analysis of 24,542 Q and As, Claude 3 and GPT-4 performed better on semantic Q and As (68.7%, n=1593 and 68.4%, n=1709), respectively. Then on numerical Q and As (61.3%, n=8583 and 56.7%, n=12,038), respectively, with Claude 3 outperforming GPT-4 in numeric accuracy ( P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622.
- Kahun Medical Ltd, Givatayim, Israel.
- Faculty of Medicine, Hebrew University of Jerusalem, Jerusalem, Israel.
- School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
- The Azrieli Faculty of Medicine, Bar-Ilan University, Safed, Israel.
- Kaplan Medical Center, Rehovot, Israel.
ORCID-Profile: Eden Avnat, Michal Levy, Daniel Herstain, Elia Yanko, Daniel Ben Joya, Michal Tzuchman Katz, Dafna Eshel, Sahar Laros, Yael Dagan, Shahar Barami, Joseph Mermelstein, Shahar Ovadia, Noam Shomron, Varda Shalev, Raja-Elie E Abdulnour
Zitieren
APA
Avnat, E., Levy, M., Herstain, D., Yanko, E., Ben Joya, D., Tzuchman Katz, M., Eshel, D., Laros, S., Dagan, Y., Barami, S., Mermelstein, J., Ovadia, S., Shomron, N., Shalev, V., & Abdulnour, R. E. (2025). Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers. Journal of Medical Internet Research. https://doi.org/10.2196/64452
Vancouver
Avnat E, Levy M, Herstain D, Yanko E, Ben Joya D, Tzuchman Katz M, et al. Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers. Journal of Medical Internet Research. 2025. doi:10.2196/64452
BibTeX
@article{Avnat2025,
author = {Avnat, Eden and Levy, Michal and Herstain, Daniel and Yanko, Elia and Ben Joya, Daniel and Tzuchman Katz, Michal and Eshel, Dafna and Laros, Sahar and Dagan, Yael and Barami, Shahar and Mermelstein, Joseph and Ovadia, Shahar and Shomron, Noam and Shalev, Varda and Abdulnour, Raja-Elie E},
title = {Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/64452}
}
EUVIMED