Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination
Journal of Orthopaedic Science · 2025
DOI: 10.1016/j.jos.2025.03.003 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Orthopaedic Science, 2025. Verfasst von Juntaro Maruyama und 16 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan. Hauptthemen laut MEDLINE: Clinical Competence, Educational Measurement, Large Language Models, Orthopedics. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Gefördert von: Japan Society for the Promotion of Science, Japan Society for the Promotion of Science London.
Abstract (Originalsprache)
INTRODUCTION: Large language models (LLMs) have shown potential in medical applications. However, their effectiveness in specialized medical domains remains underexplored. The integration of Retrieval-Augmented Generation (RAG) has been proposed to improve these models by reducing hallucinations and enhancing domain-specific information access. Through this evaluation, we aim to assess whether RAG can effectively bridge the gap between LLMs' current capabilities and the accuracy needed for medical use by examining GPT-3.5 Turbo, GPT-4o, and o1-preview on the 2024 Japanese Orthopedic Specialist Examination. METHODS: A specialized database was created using the "Standard Textbook of Orthopedics", and GPT-3.5 Turbo, GPT-4o, and o1-preview were evaluated with and without RAG. Models were tested on text-based and image-based questions exactly as presented in Japanese. An error analysis was conducted to identify key performance factors. RESULTS: GPT-3.5 Turbo showed no substantial improvement with RAG, with its overall accuracy remaining at 28 %, compared to its baseline of 29 % without RAG. GPT-4o rose from 62 % to 72 %, while o1-preview increased from 67 % to 84 %. Error analysis indicated that GPT-3.5 Turbo primarily failed to apply retrieved data, whereas GPT-4o and o1-preview made errors when the database lacked relevant information or when dealing with image-based questions. CONCLUSIONS: The integration of RAG significantly boosted performance for GPT-4o and especially o1-preview. While both models surpassed the passing threshold, o1-preview demonstrated a level of proficiency relevant to clinical practice. However, RAG did not improve performance on GPT-3.5 Turbo because it lacks effective reasoning abilities.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan.
- Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan; Center for Frontier Medical Engineering, Chiba University, Japan. Electronic address: satoshimaki@gmail.com.
- Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan; Center for Frontier Medical Engineering, Chiba University, Japan.
Förderung
- Japan Society for the Promotion of Science
- Japan Society for the Promotion of Science London
Zitieren
APA
Maruyama, J., Maki, S., Furuya, T., Nagashima, Y., Kitagawa, K., Toki, Y., Iwata, S., Yazaki, M., Kitamura, T., Gushiken, S., Noguchi, Y., Miura, M., Inoue, M., Shiga, Y., Inage, K., Orita, S., & Ohtori, S. (2025). Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination. Journal of Orthopaedic Science. https://doi.org/10.1016/j.jos.2025.03.003
Vancouver
Maruyama J, Maki S, Furuya T, Nagashima Y, Kitagawa K, Toki Y, et al. Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination. Journal of Orthopaedic Science. 2025. doi:10.1016/j.jos.2025.03.003
BibTeX
@article{Maruyama2025,
author = {Maruyama, Juntaro and Maki, Satoshi and Furuya, Takeo and Nagashima, Yuki and Kitagawa, Kyota and Toki, Yasunori and Iwata, Shuhei and Yazaki, Megumi and Kitamura, Takaki and Gushiken, Sho and Noguchi, Yuji and Miura, Masataka and Inoue, Masahiro and Shiga, Yasuhiro and Inage, Kazuhide and Orita, Sumihisa and Ohtori, Seiji},
title = {Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination},
journal = {Journal of Orthopaedic Science},
year = {2025},
doi = {10.1016/j.jos.2025.03.003}
}
EUVIMED