Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions
Anaesthesia Reports · 2026-01 · 14 (1)
DOI: 10.1002/anr3.70065 ↗Kurz gefasst
Fachartikel in der Zeitschrift Anaesthesia Reports, online erschienen am 12.05.2026. Verfasst von B Gorst und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Biological Sciences, Faculty of Biology, Medicine and Health University of Manchester Manchester UK. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Tracheostomy training is effective, yet significant face‐to‐face training barriers remain. We aimed to develop, train and test a generative artificial intelligence instructor (AI‐Brendan). AI‐Brendan utilises the base model of ChatGPT‐4.0, augmented by a curated dataset of expert‐answered tracheostomy care questions, delivered with a contextually responsive conversational interface (Inworld). AI‐Brendan's responses to five randomly selected questions were compared with ChatGPT‐4.0 and five human expert responses using a modified Ensuring Quality Information for Patients tool, rated by six blinded independent adjudicators. A total of 450 Ensuring Quality Information for Patients scores were generated from responses (150 each from AI‐Brendan, ChatGPT‐4.0 and Experts), with no alarming responses. AI‐Brendan (mean score 12.6 ± 3.0) performed significantly better than ChatGPT‐4.0 (11.4 ± 2.4) and Experts (8.9 ± 3.6, ANOVA p < 0.01). Friedman testing demonstrated no significant differences across repeated responses, indicating consistency. Within‐question variability indicated highly consistent responses. Inter‐rater reliability was good (Cronbach's α = 0.863). Adjudicator's ability to correctly determine answer source was 50% for Experts, 39% for AI‐Brendan and 33% for ChatGPT‐4.0. Our expert‐trained generative artificial intelligence produced “better” answers than open‐source ChatGPT‐4.0 and experts when rated by the modified Ensuring Quality Information for Patients tool. A validated artificial intelligence instructor has significant implications for addressing training barriers, increasing efficacy, overcoming scheduling, financial or geographical limitations, offering consistent, accessible and flexible training.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Biological Sciences, Faculty of Biology, Medicine and Health University of Manchester Manchester UK.
- Division of Informatics, Imaging and Data Sciences University of Manchester Manchester UK.
- Sentira XR St Albans Hertfordshire UK.
- North West School of Anaesthetics Liverpool UK.
- Manchester Academic Critical Care, Division of Infection, Immunity and Respiratory Medicine, School of Biological Sciences, Faculty of Biology, Medicine and Health University of Manchester, Manchester Academic Health Sc…
- Department of Anaesthesia Manchester University NHS Foundation Trust Manchester UK.
ORCID-Profile: E Johnstone, J Abbas, A Payton, B A McGrath
Zitieren
APA
Gorst, B., Johnstone, E., Abbas, J., Moran, M., Payton, A., & McGrath, B. A. (2026). Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions. Anaesthesia Reports, 14(1). https://doi.org/10.1002/anr3.70065
Vancouver
Gorst B, Johnstone E, Abbas J, Moran M, Payton A, McGrath BA. Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions. Anaesthesia Reports. 2026;14(1). doi:10.1002/anr3.70065
BibTeX
@article{Gorst2026,
author = {Gorst, B and Johnstone, E and Abbas, J and Moran, M and Payton, A and McGrath, B A},
title = {Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions},
journal = {Anaesthesia Reports},
year = {2026},
volume = {14},
number = {1},
doi = {10.1002/anr3.70065}
}
EUVIMED