ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions
JMIR Medical Education · 05.12.2023 · 9 · e49183
DOI: 10.2196/49183 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 05.12.2023. Verfasst von Christoph Raphael Buhr und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany. Die Arbeit wurde bisher 46-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs), such as ChatGPT (Open AI), are increasingly used in medicine and supplement standard search engines as information sources. This leads to more “consultations” of LLMs about personal medical symptoms.
Objective
This study aims to evaluate ChatGPT’s performance in answering clinical case–based questions in otorhinolaryngology (ORL) in comparison to ORL consultants’ answers.
Methods
We used 41 case-based questions from established ORL study books and past German state examinations for doctors. The questions were answered by both ORL consultants and ChatGPT 3. ORL consultants rated all responses, except their own, on medical adequacy, conciseness, coherence, and comprehensibility using a 6-point Likert scale. They also identified (in a blinded setting) if the answer was created by an ORL consultant or ChatGPT. Additionally, the character count was compared. Due to the rapidly evolving pace of technology, a comparison between responses generated by ChatGPT 3 and ChatGPT 4 was included to give an insight into the evolving potential of LLMs.
Results
Ratings in all categories were significantly higher for ORL consultants (P
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany.
- School of Medicine, University of St Andrews, St Andrews, United Kingdom.
- School of Computer Science, University of St Andrews, St Andrews, United Kingdom.
- Institute of Digital Medicine, Philipps-University Marburg and University Hospital of Giessen and Marburg, Marburg, Germany.
ORCID-Profile: Christoph Raphael Buhr, Harry Smith, Tilman Huppertz, Katharina Bahr-Hamm, Christoph Matthias, Andrew Blaikie, Tom Kelsey, Sebastian Kuhn, Jonas Eckrich
Zitieren
APA
Buhr, C. R., Smith, H., Huppertz, T., Bahr-Hamm, K., Matthias, C., Blaikie, A., Kelsey, T., Kuhn, S., & Eckrich, J. (2023). ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions. JMIR Medical Education, 9, e49183. https://doi.org/10.2196/49183
Vancouver
Buhr CR, Smith H, Huppertz T, Bahr-Hamm K, Matthias C, Blaikie A, et al. ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions. JMIR Medical Education. 2023;9:e49183. doi:10.2196/49183
BibTeX
@article{Buhr2023,
author = {Buhr, Christoph Raphael and Smith, Harry and Huppertz, Tilman and Bahr-Hamm, Katharina and Matthias, Christoph and Blaikie, Andrew and Kelsey, Tom and Kuhn, Sebastian and Eckrich, Jonas},
title = {ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions},
journal = {JMIR Medical Education},
year = {2023},
volume = {9},
pages = {e49183},
doi = {10.2196/49183}
}
EUVIMED