Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study
Journal of Medical Internet Research · 11.09.2026 · 28 · e94755-e94755
DOI: 10.2196/94755 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 11.09.2026. Verfasst von Linkai Li und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical Engineering, Stanford University, Stanford, CA, United States. Hauptthemen laut MEDLINE: Large Language Models, Audiology, Benchmarking, Decision Support Systems, Clinical. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) are increasingly being explored for clinical decision support, but their performance in audiology has not been systematically benchmarked using clinically grounded case materials and rubric-based safety evaluations.
Objective
This study aimed to develop and evaluate AUDIOLOGYBENCH, a 3-tier benchmark for characterizing frontier LLM capability in clinical audiology along (1) curated domain knowledge, (2) literature-derived evidence, and (3) clinical reasoning under multimodal case input, with an explicit human audit of the automated adjudicator on the primary end point.
Methods
The benchmark comprises 3139 objective items from educational resources, 3175 research article–derived items from peer-reviewed articles published between 2015 and 2025, and 67 multimodal clinical case studies graded against a standardized A-F rubric with 6 prespecified critical-error types that cap scores at D or F. Eight models were evaluated on the educational objective items: 4 frontier multimodal models (Gemini 2.5 Pro, Grok 4, OpenAI O3, and Claude Sonnet 4 Thinking) were evaluated on the research article–derived items, and on 804 case study evaluations. Adjudication used Gemini 2.5 Pro (objective and research-derived items) and Claude Opus 4.5 (case studies). The case study adjudicator was independently audited against PhD-level audiologist consensus on blinded subsamples, supplemented by a post-stratified human-calibrated sensitivity analysis.
Results
A striking task-type dissociation emerged on case studies: clinical recommendations (Q3) achieved a mean score of 89.74 (SD 13.92, 95% CI 88.07‐91.41), a 98.1% (263/268) pass rate, and no dangerous recommendations; audiometric numerical interpretation (Q1) achieved a mean score of 67.89 (SD 18.47, 95% CI 65.68‐70.10), with a 35.4% (95/268) critical-error rate; and differential diagnosis (Q2) achieved a mean score of 67.79 (SD 15.33, 95% CI 65.95‐69.63). Question type, not model selection, dominated performance (eta-squared_H=0.333 vs 0.001; rank biserial r ≥0.679). Interreviewer reliability between audiologists was high (quadratic-weighted κ of 0.78 and 0.85 across the 80-item and 50-item audits, respectively). When 2 audiologists regraded all 80 model Q1 responses with the diagnostic images available, the adjudicator’s per-item Q1 labels diverged from human judgment (κ=0.05; overflagging; sensitivity: 19/26, 73%; positive predictive value: 19/53, 36%), yet its reweighted Q1 critical-error rate (36.2%) was broadly consistent with the image-grounded human estimates (28%‐34%), suggesting no systematic inflation of the headline rate. The principal Q3>{Q1, Q2} ranking was preserved under post-stratified human calibration. Web-style multiple-choice items showed ceiling effects (>95% accuracy); short-answer prompts remained challenging (best 30%).
Conclusions
Current frontier LLMs show strong recommendation generation but substantial limitations in audiometric numerical interpretation that are shared across models and that an automated adjudicator partially miscalibrated at the per-item level. AUDIOLOGYBENCH characterizes capability boundaries rather than certifying clinical readiness. Deployment of LLM-assisted audiology workflows requires structured human verification of all numerical findings and awareness of fabrication and severity misclassification failure modes documented here.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States.
- Orka Labs Inc, Shanghai, China.
- Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC, Canada.
- Faculty of Education, University of Hong Kong, Hong Kong, Hong Kong, China (Hong Kong).
- Department of Otolaryngology-Head and Neck Surgery, Stanford University, Stanford, CA, United States.
- Department of Electrical Engineering, Stanford University, Address: McCullough Building, Room 351, 476 Lomita Mall, Stanford, CA, United States, 1 650-723-8671.
Zitieren
APA
Li, L., Mo, C., Zhou, H., Yu, H., Lu, C., Wang, S., Athreya, V. M., Fitzgerald, M. B., & Wang, S. X. (2026). Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study. Journal of Medical Internet Research, 28, e94755-e94755. https://doi.org/10.2196/94755
Vancouver
Li L, Mo C, Zhou H, Yu H, Lu C, Wang S, et al. Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study. Journal of Medical Internet Research. 2026;28:e94755-e94755. doi:10.2196/94755
BibTeX
@article{Li2026,
author = {Li, Linkai and Mo, Changgeng and Zhou, Haoshuai and Yu, Hanlin and Lu, Congxi and Wang, Shangqiguo and Athreya, Varsha M and Fitzgerald, Matthew B and Wang, Shan X},
title = {Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e94755--e94755},
doi = {10.2196/94755}
}
EUVIMED