EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating the Precision and Dependability of Medical Answers Generated by ChatGPT

Zain Abidin

Journal of Science, Technology, Education, Art and Medicine · 10.06.2024 · 1 (1) · 16-22

DOI: 10.63137/jsteam.744858 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Science, Technology, Education, Art and Medicine, online erschienen am 10.06.2024. Verfasst von Zain Abidin. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study aims to assess the accuracy and depth of ChatGPT’s responses to medical questions posed by physicians, providing preliminary evidence of its reliability in offering precise and comprehensive information. Furthermore, the study will shed light on the limitations inherent in AI-generated medical advice.

Methods

This research involved 10 physicians formulating questions for ChatGPT without patient-specific data. Approximately 29% of the 35 invited doctors participated, creating eight questions each. The questions covered easy, medium, and hard levels, with yes/no or descriptive responses. ChatGPT’s responses were evaluated by physicians for accuracy and completeness using established Likert scales. An internal validation re-submitted questions with low accuracy scores, and statistical measures analyzed the outcomes, revealing insights into response consistency and variation over time.

Results

The analysis of 80 ChatGPT-generated answers revealed a median accuracy score of 4 (mean 4.7, SD 2.6) and a median completeness score of 2 (mean 1.8, SD 1.5). Notably, 30% of responses achieved the highest accuracy score (6), and 38.7% were rated nearly all correct (5), while 8% were deemed completely incorrect (1). Inaccurate answers were more common for physician-rated hard questions. Completeness varied, with 45% considered comprehensive, 37.5% adequate, and 17.5% incomplete. Modest correlation (Spearman’s r = 0.3) existed between accuracy and completeness across all questions.

Conclusion

Integrating language models like ChatGPT in medical practice shows promise, but cautious considerations are crucial for safe use. While AI-generated responses display commendable accuracy and completeness, ongoing refinement is needed for reliability. This research lays a foundation for AI integration in healthcare, underscoring the importance of continuous evaluation and regulatory measures to ensure safe and effective implementation.

Zitieren

APA

Abidin, Z. (2024). Evaluating the Precision and Dependability of Medical Answers Generated by ChatGPT. Journal of Science, Technology, Education, Art and Medicine, 1(1), 16-22. https://doi.org/10.63137/jsteam.744858

Vancouver

Abidin Z. Evaluating the Precision and Dependability of Medical Answers Generated by ChatGPT. Journal of Science, Technology, Education, Art and Medicine. 2024;1(1):16-22. doi:10.63137/jsteam.744858

BibTeX
@article{Abidin2024,
  author = {Abidin, Zain},
  title = {Evaluating the Precision and Dependability of Medical Answers Generated by ChatGPT},
  journal = {Journal of Science, Technology, Education, Art and Medicine},
  year = {2024},
  volume = {1},
  number = {1},
  pages = {16--22},
  doi = {10.63137/jsteam.744858}
}
Kostenlos starten →