Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions
CHEST · 2026-05 · 169 (5) · 1286-1297
DOI: 10.1016/j.chest.2025.11.034 ↗Kurz gefasst
Fachartikel in der Zeitschrift CHEST, 2026-05. Verfasst von Pei Ye Li und 2 weiteren Autor:innen. Beteiligte Einrichtung: Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada. Hauptthemen laut MEDLINE: Asthma, Educational Measurement, Language. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 42 Literaturangaben.
Abstract (Originalsprache)
BACKGROUND: Large language models (LLMs) are demonstrating increasing promise across clinical applications, but their domain-specific knowledge in asthma has not been thoroughly explored. Additionally, state-of-the-art models released in 2025 (ChatGPT-5, ChatGPT-o3, Claude-3.7, DeepSeek-V3, and Grok-3) have yet to be studied in asthma. RESEARCH QUESTION: How accurate are LLMs at answering adult asthma multiple choice questions (MCQs) and Objective Structured Clinical Examinations (OSCEs)? STUDY DESIGN AND METHODS: Fourteen LLM chatbots (Asthma-GPT, ChatGPT-4o, ChatGPT-o1, ChatGPT-o3-mini, ChatGPT-o3-mini-high, ChatGPT-5, Claude-3.7-Sonnet, DeepSeek-V3, Gemini-2.0-Flash, Grok-3, Le-Chat, Llama-3, NICE-Asthma-Bot, and Biomistral) were tested in 5 iterations on 116 adult asthma MCQs and 3 asthma OSCEs. Using generalized linear mixed-effects models, we compared accuracy between the LLMs using log-odds scale and pairwise comparisons with Tukey adjustment, and assessed difference in MCQs vs OSCEs, generic vs medicine-specific models, open-source vs proprietary models, and patient- vs clinician-oriented questions. RESULTS: MCQ accuracy was excellent, with 13 of 14 models exceeding 85%, and ChatGPT-o1, Asthma-GPT, NICE-Asthma-Bot, ChatGPT-5, and ChatGPT-4o surpassing 95% accuracy. OSCE accuracy varied more significantly, but most models still achieved 70% to 86% accuracy. ChatGPT-o1 ranked highest in MCQs, and ChatGPT-5 ranked highest in OSCEs. Generalized linear mixed-effects model revealed that LLMs performed better at MCQs than OSCEs (92.8% vs 71.1%, P < .001), and that LLMs performed better at patient-oriented MCQs than clinician-oriented MCQs (96.3% vs 85.6%, P < .01). No significant differences were found between open-source and proprietary models in MCQs, or between generic and medicine-specific LLMs. INTERPRETATION: Our results indicate that current LLMs demonstrate high accuracy across asthma-related clinical and patient-oriented questions, particularly the ChatGPT family of models. These models may be useful tools to integrate into future patient and clinician education interventions.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
- Division of Respirology, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.
- Division of Respirology, St. Michael's Hospital, Toronto, ON, Canada; Department of Medicine, University of Toronto, Toronto, ON, Canada; Division of Respirology, Women's College Hospital, Toronto, ON, Canada. Electroni…
Zitieren
APA
Li, P. Y., Gershon, A., & Kouri, A. (2026). Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions. CHEST, 169(5), 1286-1297. https://doi.org/10.1016/j.chest.2025.11.034
Vancouver
Li PY, Gershon A, Kouri A. Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions. CHEST. 2026;169(5):1286-1297. doi:10.1016/j.chest.2025.11.034
BibTeX
@article{Li2026,
author = {Li, Pei Ye and Gershon, Andrea and Kouri, Andrew},
title = {Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions},
journal = {CHEST},
year = {2026},
volume = {169},
number = {5},
pages = {1286--1297},
doi = {10.1016/j.chest.2025.11.034}
}
EUVIMED