The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland
Clinical Kidney Journal · 01.08.2024 · 17 (8)
DOI: 10.1093/ckj/sfae193 ↗Kurz gefasst
Fachartikel in der Zeitschrift Clinical Kidney Journal, online erschienen am 22.06.2024. Verfasst von Jan Nicikowski und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of Zielona Gora, Faculty of Medicine and Health Sciences, Student Scientific Section of Clinical Nutrition, Zielona Góra, Poland. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Ministry of Science and Higher Education.
Abstract (Originalsprache)
Background
In November 2022, OpenAI released a chatbot named ChatGPT, a product capable of processing natural language to create human-like conversational dialogue. It has generated a lot of interest, including from the scientific community and the medical science community. Recent publications have shown that ChatGPT can correctly answer questions from medical exams such as the United States Medical Licensing Examination and other specialty exams. To date, there have been no studies in which ChatGPT has been tested on specialty questions in the field of nephrology anywhere in the world.
Methods
Using the ChatGPT-3.5 and -4.0 algorithms in this comparative cross-sectional study, we analysed 1560 single-answer questions from the national specialty exam in nephrology from 2017 to 2023 that were available in the Polish Medical Examination Center's question database along with answer keys.
Results
Of the 1556 questions posed to ChatGPT-4.0, correct answers were obtained with an accuracy of 69.84%, compared with ChatGPT-3.5 (45.70%, P = .0001) and with the top results of medical doctors (85.73%, P = .0001). Of the 13 tests, ChatGPT-4.0 exceeded the required ≥60% pass rate in 11 tests passed, and scored higher than the average of the human exam results.
Conclusion
ChatGPT-3.5 was not spectacularly successful in nephrology exams. The ChatGPT-4.0 algorithm was able to pass most of the analysed nephrology specialty exams. New generations of ChatGPT achieve similar results to humans. The best results of humans are better than those of ChatGPT-4.0.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University of Zielona Gora, Faculty of Medicine and Health Sciences, Student Scientific Section of Clinical Nutrition, Zielona Góra, Poland.
- University of Zielona Góra, Faculty of Medicine and Health Sciences, Department of Anaesthesiology, Intensive Care and Emergency Medicine, Zielona Góra, Poland.
- Poznan University of Medical Sciences, Department of General and Transplant Surgery, Poznan, Poland.
ORCID-Profile: Jan Nicikowski, Mikołaj Szczepański, Miłosz Miedziaszczyk, Bartosz Kudliński
Förderung
- Ministry of Science and Higher Education
Zitieren
APA
Nicikowski, J., Szczepański, M., Miedziaszczyk, M., & Kudliński, B. (2024). The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clinical Kidney Journal, 17(8). https://doi.org/10.1093/ckj/sfae193
Vancouver
Nicikowski J, Szczepański M, Miedziaszczyk M, Kudliński B. The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clinical Kidney Journal. 2024;17(8). doi:10.1093/ckj/sfae193
BibTeX
@article{Nicikowski2024,
author = {Nicikowski, Jan and Szczepański, Mikołaj and Miedziaszczyk, Miłosz and Kudliński, Bartosz},
title = {The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland},
journal = {Clinical Kidney Journal},
year = {2024},
volume = {17},
number = {8},
doi = {10.1093/ckj/sfae193}
}
EUVIMED