EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers

Eden Avnat iD, Michal Levy iD, Daniel Herstain iD, Elia Yanko iD, Daniel Ben Joya iD, Michal Tzuchman Katz iD, Dafna Eshel iD, Sahar Laros iD, Yael Dagan iD, Shahar Barami iD, Joseph Mermelstein iD, Shahar Ovadia iD und 3 weitere

Journal of Medical Internet Research · 2025

DOI: 10.2196/64452 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Eden Avnat und 14 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622. Hauptthemen laut MEDLINE: Semantics, Benchmarking, Evidence-Based Medicine, Language. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical problem-solving requires processing of semantic medical knowledge, such as illness scripts, and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate nonlanguage evidence-based answers to clinical questions is inherently limited by tokenization.

Objective

This study aimed to evaluate LLMs’ performance on two question types: numeric (correlating findings) and semantic (differentiating entities), while examining differences within and between LLMs in medical aspects and comparing their performance to humans.

Methods

To generate straightforward multichoice questions and answers (Q and As) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (containing data from more than 50,000 peer-reviewed studies) and created the EBM questions and answers (EBMQAs). EBMQA comprises 105,222 Q and As, categorized by medical topics (eg, medical disciplines) and nonmedical topics (eg, question length), and classified into numerical or semantic types. We benchmarked a dataset of 24,000 Q and As on two state-of-the-art LLMs, GPT-4 (OpenAI) and Claude 3 Opus (Anthropic). We evaluated the LLM’s accuracy on semantic and numerical question types and according to sublabeled topics. In addition, we examined the question-answering rate of LLMs by enabling them to choose to abstain from responding to questions. For validation, we compared the results for 100 unrelated numerical EBMQA questions between six human medical experts and the two language models.

Results

In an analysis of 24,542 Q and As, Claude 3 and GPT-4 performed better on semantic Q and As (68.7%, n=1593 and 68.4%, n=1709), respectively. Then on numerical Q and As (61.3%, n=8583 and 56.7%, n=12,038), respectively, with Claude 3 outperforming GPT-4 in numeric accuracy ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SemanticsBenchmarkingEvidence-Based MedicineLanguageHumansCross-Sectional StudiesLarge Language Models

Stichwörter der Autor:innen

benchmarkdatasetevidence-based medicinelarge language modelsquestions and answers

Autor:innen und Institutionen

  • Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622.
  • Kahun Medical Ltd, Givatayim, Israel.
  • Faculty of Medicine, Hebrew University of Jerusalem, Jerusalem, Israel.
  • School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
  • The Azrieli Faculty of Medicine, Bar-Ilan University, Safed, Israel.
  • Kaplan Medical Center, Rehovot, Israel.

ORCID-Profile: Eden Avnat, Michal Levy, Daniel Herstain, Elia Yanko, Daniel Ben Joya, Michal Tzuchman Katz, Dafna Eshel, Sahar Laros, Yael Dagan, Shahar Barami, Joseph Mermelstein, Shahar Ovadia, Noam Shomron, Varda Shalev, Raja-Elie E Abdulnour

Zitieren

APA

Avnat, E., Levy, M., Herstain, D., Yanko, E., Ben Joya, D., Tzuchman Katz, M., Eshel, D., Laros, S., Dagan, Y., Barami, S., Mermelstein, J., Ovadia, S., Shomron, N., Shalev, V., & Abdulnour, R. E. (2025). Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers. Journal of Medical Internet Research. https://doi.org/10.2196/64452

Vancouver

Avnat E, Levy M, Herstain D, Yanko E, Ben Joya D, Tzuchman Katz M, et al. Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers. Journal of Medical Internet Research. 2025. doi:10.2196/64452

BibTeX
@article{Avnat2025,
  author = {Avnat, Eden and Levy, Michal and Herstain, Daniel and Yanko, Elia and Ben Joya, Daniel and Tzuchman Katz, Michal and Eshel, Dafna and Laros, Sahar and Dagan, Yael and Barami, Shahar and Mermelstein, Joseph and Ovadia, Shahar and Shomron, Noam and Shalev, Varda and Abdulnour, Raja-Elie E},
  title = {Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/64452}
}
Kostenlos starten →