EUVIMEDEUVIMED entdecken
Crossref

Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions

Pei Ye Li, Andrea Gershon, Andrew Kouri

CHEST · 2026-05 · 169 (5) · 1286-1297

DOI: 10.1016/j.chest.2025.11.034 ↗

Kurz gefasst

Fachartikel in der Zeitschrift CHEST, 2026-05. Verfasst von Pei Ye Li und 2 weiteren Autor:innen. Beteiligte Einrichtung: Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada. Hauptthemen laut MEDLINE: Asthma, Educational Measurement, Language. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 42 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

BACKGROUND: Large language models (LLMs) are demonstrating increasing promise across clinical applications, but their domain-specific knowledge in asthma has not been thoroughly explored. Additionally, state-of-the-art models released in 2025 (ChatGPT-5, ChatGPT-o3, Claude-3.7, DeepSeek-V3, and Grok-3) have yet to be studied in asthma. RESEARCH QUESTION: How accurate are LLMs at answering adult asthma multiple choice questions (MCQs) and Objective Structured Clinical Examinations (OSCEs)? STUDY DESIGN AND METHODS: Fourteen LLM chatbots (Asthma-GPT, ChatGPT-4o, ChatGPT-o1, ChatGPT-o3-mini, ChatGPT-o3-mini-high, ChatGPT-5, Claude-3.7-Sonnet, DeepSeek-V3, Gemini-2.0-Flash, Grok-3, Le-Chat, Llama-3, NICE-Asthma-Bot, and Biomistral) were tested in 5 iterations on 116 adult asthma MCQs and 3 asthma OSCEs. Using generalized linear mixed-effects models, we compared accuracy between the LLMs using log-odds scale and pairwise comparisons with Tukey adjustment, and assessed difference in MCQs vs OSCEs, generic vs medicine-specific models, open-source vs proprietary models, and patient- vs clinician-oriented questions. RESULTS: MCQ accuracy was excellent, with 13 of 14 models exceeding 85%, and ChatGPT-o1, Asthma-GPT, NICE-Asthma-Bot, ChatGPT-5, and ChatGPT-4o surpassing 95% accuracy. OSCE accuracy varied more significantly, but most models still achieved 70% to 86% accuracy. ChatGPT-o1 ranked highest in MCQs, and ChatGPT-5 ranked highest in OSCEs. Generalized linear mixed-effects model revealed that LLMs performed better at MCQs than OSCEs (92.8% vs 71.1%, P < .001), and that LLMs performed better at patient-oriented MCQs than clinician-oriented MCQs (96.3% vs 85.6%, P < .01). No significant differences were found between open-source and proprietary models in MCQs, or between generic and medicine-specific LLMs. INTERPRETATION: Our results indicate that current LLMs demonstrate high accuracy across asthma-related clinical and patient-oriented questions, particularly the ChatGPT family of models. These models may be useful tools to integrate into future patient and clinician education interventions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AsthmaEducational MeasurementLanguageHumansAdultMaleFemaleSurveys and QuestionnairesLarge Language Models

Stichwörter der Autor:innen

artificial intelligenceasthmalarge language model

Autor:innen und Institutionen

  • Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
  • Division of Respirology, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.
  • Division of Respirology, St. Michael's Hospital, Toronto, ON, Canada; Department of Medicine, University of Toronto, Toronto, ON, Canada; Division of Respirology, Women's College Hospital, Toronto, ON, Canada. Electroni…

Zitieren

APA

Li, P. Y., Gershon, A., & Kouri, A. (2026). Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions. CHEST, 169(5), 1286-1297. https://doi.org/10.1016/j.chest.2025.11.034

Vancouver

Li PY, Gershon A, Kouri A. Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions. CHEST. 2026;169(5):1286-1297. doi:10.1016/j.chest.2025.11.034

BibTeX
@article{Li2026,
  author = {Li, Pei Ye and Gershon, Andrea and Kouri, Andrew},
  title = {Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions},
  journal = {CHEST},
  year = {2026},
  volume = {169},
  number = {5},
  pages = {1286--1297},
  doi = {10.1016/j.chest.2025.11.034}
}
Kostenlos starten →