EUVIMEDEUVIMED entdecken
Open Access

ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions

Christoph Raphael Buhr iD, Harry Smith iD, Tilman Huppertz iD, Katharina Bahr-Hamm iD, Christoph Matthias iD, Andrew Blaikie iD, Tom Kelsey iD, Sebastian Kuhn iD, Jonas Eckrich iD

JMIR Medical Education · 05.12.2023 · 9 · e49183

DOI: 10.2196/49183 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 05.12.2023. Verfasst von Christoph Raphael Buhr und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany. Die Arbeit wurde bisher 46-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs), such as ChatGPT (Open AI), are increasingly used in medicine and supplement standard search engines as information sources. This leads to more “consultations” of LLMs about personal medical symptoms.

Objective

This study aims to evaluate ChatGPT’s performance in answering clinical case–based questions in otorhinolaryngology (ORL) in comparison to ORL consultants’ answers.

Methods

We used 41 case-based questions from established ORL study books and past German state examinations for doctors. The questions were answered by both ORL consultants and ChatGPT 3. ORL consultants rated all responses, except their own, on medical adequacy, conciseness, coherence, and comprehensibility using a 6-point Likert scale. They also identified (in a blinded setting) if the answer was created by an ORL consultant or ChatGPT. Additionally, the character count was compared. Due to the rapidly evolving pace of technology, a comparison between responses generated by ChatGPT 3 and ChatGPT 4 was included to give an insight into the evolving potential of LLMs.

Results

Ratings in all categories were significantly higher for ORL consultants (P

Schlagwörter

Stichwörter der Autor:innen

AIChatGPTLLMLLMsORLartificial intelligencechatbotchatbotsdigital healthglobal healthlanguage modellarge language modelslow- and middle-income countriesotorhinolaryngologytelehealth

Autor:innen und Institutionen

  • Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany.
  • School of Medicine, University of St Andrews, St Andrews, United Kingdom.
  • School of Computer Science, University of St Andrews, St Andrews, United Kingdom.
  • Institute of Digital Medicine, Philipps-University Marburg and University Hospital of Giessen and Marburg, Marburg, Germany.

ORCID-Profile: Christoph Raphael Buhr, Harry Smith, Tilman Huppertz, Katharina Bahr-Hamm, Christoph Matthias, Andrew Blaikie, Tom Kelsey, Sebastian Kuhn, Jonas Eckrich

Zitieren

APA

Buhr, C. R., Smith, H., Huppertz, T., Bahr-Hamm, K., Matthias, C., Blaikie, A., Kelsey, T., Kuhn, S., & Eckrich, J. (2023). ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions. JMIR Medical Education, 9, e49183. https://doi.org/10.2196/49183

Vancouver

Buhr CR, Smith H, Huppertz T, Bahr-Hamm K, Matthias C, Blaikie A, et al. ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions. JMIR Medical Education. 2023;9:e49183. doi:10.2196/49183

BibTeX
@article{Buhr2023,
  author = {Buhr, Christoph Raphael and Smith, Harry and Huppertz, Tilman and Bahr-Hamm, Katharina and Matthias, Christoph and Blaikie, Andrew and Kelsey, Tom and Kuhn, Sebastian and Eckrich, Jonas},
  title = {ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions},
  journal = {JMIR Medical Education},
  year = {2023},
  volume = {9},
  pages = {e49183},
  doi = {10.2196/49183}
}
Kostenlos starten →