EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland

Jan Nicikowski iD, Mikołaj Szczepański iD, Miłosz Miedziaszczyk iD, Bartosz Kudliński iD

Clinical Kidney Journal · 01.08.2024 · 17 (8)

DOI: 10.1093/ckj/sfae193 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Clinical Kidney Journal, online erschienen am 22.06.2024. Verfasst von Jan Nicikowski und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of Zielona Gora, Faculty of Medicine and Health Sciences, Student Scientific Section of Clinical Nutrition, Zielona Góra, Poland. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Ministry of Science and Higher Education.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

In November 2022, OpenAI released a chatbot named ChatGPT, a product capable of processing natural language to create human-like conversational dialogue. It has generated a lot of interest, including from the scientific community and the medical science community. Recent publications have shown that ChatGPT can correctly answer questions from medical exams such as the United States Medical Licensing Examination and other specialty exams. To date, there have been no studies in which ChatGPT has been tested on specialty questions in the field of nephrology anywhere in the world.

Methods

Using the ChatGPT-3.5 and -4.0 algorithms in this comparative cross-sectional study, we analysed 1560 single-answer questions from the national specialty exam in nephrology from 2017 to 2023 that were available in the Polish Medical Examination Center's question database along with answer keys.

Results

Of the 1556 questions posed to ChatGPT-4.0, correct answers were obtained with an accuracy of 69.84%, compared with ChatGPT-3.5 (45.70%, P = .0001) and with the top results of medical doctors (85.73%, P = .0001). Of the 13 tests, ChatGPT-4.0 exceeded the required ≥60% pass rate in 11 tests passed, and scored higher than the average of the human exam results.

Conclusion

ChatGPT-3.5 was not spectacularly successful in nephrology exams. The ChatGPT-4.0 algorithm was able to pass most of the analysed nephrology specialty exams. New generations of ChatGPT achieve similar results to humans. The best results of humans are better than those of ChatGPT-4.0.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTartificial intelligenceclinical nephrologymachine learningmedical education

Autor:innen und Institutionen

  • University of Zielona Gora, Faculty of Medicine and Health Sciences, Student Scientific Section of Clinical Nutrition, Zielona Góra, Poland.
  • University of Zielona Góra, Faculty of Medicine and Health Sciences, Department of Anaesthesiology, Intensive Care and Emergency Medicine, Zielona Góra, Poland.
  • Poznan University of Medical Sciences, Department of General and Transplant Surgery, Poznan, Poland.

ORCID-Profile: Jan Nicikowski, Mikołaj Szczepański, Miłosz Miedziaszczyk, Bartosz Kudliński

Förderung

  • Ministry of Science and Higher Education

Zitieren

APA

Nicikowski, J., Szczepański, M., Miedziaszczyk, M., & Kudliński, B. (2024). The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clinical Kidney Journal, 17(8). https://doi.org/10.1093/ckj/sfae193

Vancouver

Nicikowski J, Szczepański M, Miedziaszczyk M, Kudliński B. The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clinical Kidney Journal. 2024;17(8). doi:10.1093/ckj/sfae193

BibTeX
@article{Nicikowski2024,
  author = {Nicikowski, Jan and Szczepański, Mikołaj and Miedziaszczyk, Miłosz and Kudliński, Bartosz},
  title = {The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland},
  journal = {Clinical Kidney Journal},
  year = {2024},
  volume = {17},
  number = {8},
  doi = {10.1093/ckj/sfae193}
}
Kostenlos starten →