EUVIMEDDescubrir EUVIMED
Open access

Evaluating Retrieval-Augmented Large Language Models on Anesthesiology Board-Style Questions: Benchmark Study

Nguyen Quang Phuong iD, Shanq-Jang Ruan iD, Pei-Fu Chen iD

JMIR Formative Research · 2026-08-11 · 10 · e97902-e97902

DOI: 10.2196/97902 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2026-08-11. Written by Nguyen Quang Phuong and 2 co-authors. Institution: Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan. Major MEDLINE topics: Large Language Models, Benchmarking, Anesthesiology. 34 references. The full text is freely available in PubMed Central.

ResumenSubjectsDatos bibliográficosCiteRelated studies

Abstract (original language)

Background

Open-source, mid-scale large language models (LLMs) have emerged as scalable, privacy-preserving alternatives to ultra-large foundation models (eg, GPT-4) in health care systems. Techniques such as retrieval-augmented generation (RAG) enable sub-100-billion-parameter models to address highly specialized medical domains such as anesthesiology. However, studies evaluating RAG architectures on complex medical examinations remain scarce, highlighting the need for rigorous benchmarking to bridge the gap between raw parametric knowledge and clinically relevant application.

Objective

This study aimed to systematically evaluate RAG pipelines for answering anesthesiology board-style questions, quantify the effects of key design choices including hyperparameter settings, embedding models, source complexity, and chunking strategies, and compare the performance of reasoning-oriented models with that of conventional LLMs.

Methods

We conducted large-scale benchmarking using American Board of Anesthesiology-style multiple-choice questions to compare multiple RAG-enabled configurations with matched standalone LLM baselines. Configurations were first optimized on a 46-item diagnostic set and then validated on a 350-item corpus. Additional experiments on three 100-question subsets derived from the 350-item corpus were used to assess the effects of source selection, source complexity, information density, and chunking strategy on answer accuracy. Models including Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Llama-3.3-70B-Instruct, Qwen2.5-7B and Qwen2.5-72B, and Qwen3-8B and Qwen3-32B reasoning models were evaluated under this framework. Self-reflective RAG (self-RAG) with adaptive retrieval techniques was also implemented and evaluated. Cochran Q and McNemar tests were used to assess performance differences across configurations and model pairs.

Results

The RAG framework increased the number of correct answers. System stability peaked under highly deterministic sampling configurations (temperature=0.1, top-p [nucleus sampling]=0.1). High-capacity general-text embeddings and applying context-preserving semantic chunking further improved accuracy. Standard RAG provided only modest gains over nonaugmented baselines, improving accuracy from 50.29% to 56.57%, and self-RAG yielded similarly limited gains of up to 4.85 percentage points. Overall, the Qwen family outperformed the Llama series. The 32-billion-parameter reasoning model Qwen-3-32B achieved an 89% correct ratio under complex distractor-heavy retrieval conditions and up to 96% with direct context, significantly outperforming the much larger 72-billion-parameter conventional model Qwen-2.5-72B-Instruct (84%). Smaller reasoning models also showed greater robustness to noise or suboptimal retrieved documents than larger conventional LLMs. Within the Llama family, increasing parameter size to 70 billion did not produce proportional performance gains on this benchmark.

Conclusions

RAG-based LLM systems improved performance on anesthesiology board-style questions, but gains depended strongly on retrieval design. Careful optimization of retrieval settings, embeddings, and chunking strategies improved robustness and answer accuracy. Reasoning-oriented models demonstrated that multistep reasoning can, in some settings, compensate for larger parameter scale. These findings provide a methodological foundation for developing locally deployable LLM systems for anesthesiology education within structured examination settings.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsBenchmarkingAnesthesiologyHumansSpecialty Boards

Author keywords

anesthesiology board examinationembedding modellarge language modelreasoning modelretrieval-augmented generation

Authors and institutions

  • Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan.
  • Department of Anesthesia, University of Iowa, 200 Hawkins Drive, 6 JCP, Iowa City, IA, 52242, United States, 1 319-356-2502.

ORCID profiles: Nguyen Quang Phuong, Shanq-Jang Ruan, Pei-Fu Chen

Cite

APA

Phuong, N. Q., Ruan, S. J., & Chen, P. F. (2026). Evaluating Retrieval-Augmented Large Language Models on Anesthesiology Board-Style Questions: Benchmark Study. JMIR Formative Research, 10, e97902-e97902. https://doi.org/10.2196/97902

Vancouver

Phuong NQ, Ruan SJ, Chen PF. Evaluating Retrieval-Augmented Large Language Models on Anesthesiology Board-Style Questions: Benchmark Study. JMIR Formative Research. 2026;10:e97902-e97902. doi:10.2196/97902

BibTeX
@article{Phuong2026,
  author = {Phuong, Nguyen Quang and Ruan, Shanq-Jang and Chen, Pei-Fu},
  title = {Evaluating Retrieval-Augmented Large Language Models on Anesthesiology Board-Style Questions: Benchmark Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e97902--e97902},
  doi = {10.2196/97902}
}
Start for free →