EUVIMEDEUVIMED entdecken
Crossref

Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination

Juntaro Maruyama, Satoshi Maki, Takeo Furuya, Yuki Nagashima, Kyota Kitagawa, Yasunori Toki, Shuhei Iwata, Megumi Yazaki, Takaki Kitamura, Sho Gushiken, Yuji Noguchi, Masataka Miura und 5 weitere

Journal of Orthopaedic Science · 2025

DOI: 10.1016/j.jos.2025.03.003 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Orthopaedic Science, 2025. Verfasst von Juntaro Maruyama und 16 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan. Hauptthemen laut MEDLINE: Clinical Competence, Educational Measurement, Large Language Models, Orthopedics. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Gefördert von: Japan Society for the Promotion of Science, Japan Society for the Promotion of Science London.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

INTRODUCTION: Large language models (LLMs) have shown potential in medical applications. However, their effectiveness in specialized medical domains remains underexplored. The integration of Retrieval-Augmented Generation (RAG) has been proposed to improve these models by reducing hallucinations and enhancing domain-specific information access. Through this evaluation, we aim to assess whether RAG can effectively bridge the gap between LLMs' current capabilities and the accuracy needed for medical use by examining GPT-3.5 Turbo, GPT-4o, and o1-preview on the 2024 Japanese Orthopedic Specialist Examination. METHODS: A specialized database was created using the "Standard Textbook of Orthopedics", and GPT-3.5 Turbo, GPT-4o, and o1-preview were evaluated with and without RAG. Models were tested on text-based and image-based questions exactly as presented in Japanese. An error analysis was conducted to identify key performance factors. RESULTS: GPT-3.5 Turbo showed no substantial improvement with RAG, with its overall accuracy remaining at 28 %, compared to its baseline of 29 % without RAG. GPT-4o rose from 62 % to 72 %, while o1-preview increased from 67 % to 84 %. Error analysis indicated that GPT-3.5 Turbo primarily failed to apply retrieved data, whereas GPT-4o and o1-preview made errors when the database lacked relevant information or when dealing with image-based questions. CONCLUSIONS: The integration of RAG significantly boosted performance for GPT-4o and especially o1-preview. While both models surpassed the passing threshold, o1-preview demonstrated a level of proficiency relevant to clinical practice. However, RAG did not improve performance on GPT-3.5 Turbo because it lacks effective reasoning abilities.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical CompetenceEducational MeasurementLarge Language ModelsOrthopedicsSpecialty BoardsJapan

Stichwörter der Autor:innen

Large language modelOrthopedic specialist examinationRetrieval-augmented generation

Autor:innen und Institutionen

  • Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan.
  • Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan; Center for Frontier Medical Engineering, Chiba University, Japan. Electronic address: satoshimaki@gmail.com.
  • Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan; Center for Frontier Medical Engineering, Chiba University, Japan.

Förderung

  • Japan Society for the Promotion of Science
  • Japan Society for the Promotion of Science London

Zitieren

APA

Maruyama, J., Maki, S., Furuya, T., Nagashima, Y., Kitagawa, K., Toki, Y., Iwata, S., Yazaki, M., Kitamura, T., Gushiken, S., Noguchi, Y., Miura, M., Inoue, M., Shiga, Y., Inage, K., Orita, S., & Ohtori, S. (2025). Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination. Journal of Orthopaedic Science. https://doi.org/10.1016/j.jos.2025.03.003

Vancouver

Maruyama J, Maki S, Furuya T, Nagashima Y, Kitagawa K, Toki Y, et al. Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination. Journal of Orthopaedic Science. 2025. doi:10.1016/j.jos.2025.03.003

BibTeX
@article{Maruyama2025,
  author = {Maruyama, Juntaro and Maki, Satoshi and Furuya, Takeo and Nagashima, Yuki and Kitagawa, Kyota and Toki, Yasunori and Iwata, Shuhei and Yazaki, Megumi and Kitamura, Takaki and Gushiken, Sho and Noguchi, Yuji and Miura, Masataka and Inoue, Masahiro and Shiga, Yasuhiro and Inage, Kazuhide and Orita, Sumihisa and Ohtori, Seiji},
  title = {Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination},
  journal = {Journal of Orthopaedic Science},
  year = {2025},
  doi = {10.1016/j.jos.2025.03.003}
}
Kostenlos starten →