Advancements in large language model accuracy for answering physical medicine and rehabilitation board review questions
PM&R · 2025-09 · 17 (9) · 1091-1096
DOI: 10.1002/pmrj.13386 ↗Kurz gefasst
Fachartikel in der Zeitschrift PM&R, online erschienen am 02.05.2025. Verfasst von Jason Bitterman und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Physical Medicine and Rehabilitation, Hartford Healthcare Medical Group, Hartford, Connecticut, USA. Hauptthemen laut MEDLINE: Physical and Rehabilitation Medicine, Language, Machine Learning, Artificial Intelligence. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 24 Literaturangaben.
Abstract (Originalsprache)
Background
There have been significant advances in machine learning and artificial intelligence technology over the past few years, leading to the release of large language models (LLMs) such as ChatGPT. There are many potential applications for LLMs in health care, but it is critical to first determine how accurate LLMs are before putting them into practice. No studies have evaluated the accuracy and precision of LLMs in responding to questions related to the field of physical medicine and rehabilitation (PM&R).
Objective
To determine the accuracy and precision of two OpenAI LLMs (GPT‐3.5, released in November 2022, and GPT‐4o, released in May 2024) in answering questions related to PM&R knowledge.
Design
Cross‐sectional study. Both LLMs were tested on the same 744 PM&R knowledge questions that covered all aspects of the field (general rehabilitation, stroke, traumatic brain injury, spinal cord injury, musculoskeletal medicine, pain medicine, electrodiagnostic medicine, pediatric rehabilitation, prosthetics and orthotics, rheumatology, and pharmacology). Each LLM was tested three times on the same question set to assess for precision.
Setting
N/A. Patients N/A.
Interventions
N/A. Main Outcome Measure Percentage of correctly answered questions.
Results
For three runs of the 744‐question set, GPT‐3.5 answered 56.3%, 56.5%, and 56.9% of the questions correctly. For three runs of the same question set, GPT‐4o answered 83.6%, 84%, and 84.1% of the questions correctly. GPT‐4o outperformed GPT‐3.5 in all subcategories of PM&R questions.
Conclusions
LLM technology is rapidly advancing, with the more recent GPT‐4o model performing much better on PM&R knowledge questions compared to GPT‐3.5. There is potential for LLMs in augmenting clinical practice, medical training, and patient education. However, the technology has limitations and physicians should remain cautious in using it in practice at this time.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Division of Physical Medicine and Rehabilitation, Hartford Healthcare Medical Group, Hartford, Connecticut, USA.
- Nebraska Medicine Department of Physical Medicine and Rehabilitation, University of Nebraska Medical Center, Omaha, Nebraska, USA.
- Department of Orthopedics and Physical Medicine, Division of Physical Medicine and Rehabilitation, Medical University of South Carolina (MUSC), Charleston, South Carolina, USA.
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh Medical Center (UPMC), Pittsburgh, Pennsylvania, USA.
ORCID-Profile: Jason Bitterman, James E Eubanks
Zitieren
APA
Bitterman, J., D'Angelo, A., Holachek, A., & Eubanks, J. E. (2025). Advancements in large language model accuracy for answering physical medicine and rehabilitation board review questions. PM&R, 17(9), 1091-1096. https://doi.org/10.1002/pmrj.13386
Vancouver
Bitterman J, D'Angelo A, Holachek A, Eubanks JE. Advancements in large language model accuracy for answering physical medicine and rehabilitation board review questions. PM&R. 2025;17(9):1091-1096. doi:10.1002/pmrj.13386
BibTeX
@article{Bitterman2025,
author = {Bitterman, Jason and D'Angelo, Alexander and Holachek, Alexandra and Eubanks, James E},
title = {Advancements in large language model accuracy for answering physical medicine and rehabilitation board review questions},
journal = {PM&R},
year = {2025},
volume = {17},
number = {9},
pages = {1091--1096},
doi = {10.1002/pmrj.13386}
}
EUVIMED