EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center

Adam Hulman iD, Ole Lindgård Dollerup iD, Jesper Friis Mortensen iD, Matthew E Fenech iD, Kasper Norman, Henrik Støvring iD, Troels Krarup Hansen iD

PLOS ONE · 31.08.2023 · 18 (8) · e0290773

DOI: 10.1371/journal.pone.0290773 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, online erschienen am 31.08.2023. Verfasst von Adam Hulman und 6 weiteren Autor:innen. Beteiligte Einrichtung: Steno Diabetes Center Aarhus, Aarhus University Hospital, Aarhus, Denmark. Hauptthemen laut MEDLINE: Diabetes Mellitus. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Novo Nordisk Fonden.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users’ 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Diabetes MellitusHumansData CollectionCluster AnalysisLanguageDenmark

Autor:innen und Institutionen

  • Steno Diabetes Center Aarhus, Aarhus University Hospital, Aarhus, Denmark.
  • Department of Public Health, Aarhus University, Aarhus, Denmark.
  • Una Health GmbH, Hamburg, Germany.
  • Department of Public Health, University of Southern Denmark, Odense, Denmark.
  • Department of Clinical Medicine, Aarhus University, Aarhus, Denmark.

ORCID-Profile: Adam Hulman, Ole Lindgård Dollerup, Jesper Friis Mortensen, Matthew E Fenech, Henrik Støvring, Troels Krarup Hansen

Förderung

  • Novo Nordisk Fonden

Zitieren

APA

Hulman, A., Dollerup, O. L., Mortensen, J. F., Fenech, M. E., Norman, K., Støvring, H., & Hansen, T. K. (2023). ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center. PLOS ONE, 18(8), e0290773. https://doi.org/10.1371/journal.pone.0290773

Vancouver

Hulman A, Dollerup OL, Mortensen JF, Fenech ME, Norman K, Støvring H, et al. ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center. PLOS ONE. 2023;18(8):e0290773. doi:10.1371/journal.pone.0290773

BibTeX
@article{Hulman2023,
  author = {Hulman, Adam and Dollerup, Ole Lindgård and Mortensen, Jesper Friis and Fenech, Matthew E and Norman, Kasper and Støvring, Henrik and Hansen, Troels Krarup},
  title = {ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center},
  journal = {PLOS ONE},
  year = {2023},
  volume = {18},
  number = {8},
  pages = {e0290773},
  doi = {10.1371/journal.pone.0290773}
}
Kostenlos starten →