EUVIMEDEUVIMED entdecken
Open Access

Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis

Xu Liu iD, Chaoli Duan iD, Min-Kyu Kim iD, Lu Zhang iD, Eunjin Jee iD, Beenu Maharjan iD, Yuwei Huang iD, Dan Du iD, Xian Jiang iD

JMIR Medical Informatics · 06.08.2024 · 12 · e59273

DOI: 10.2196/59273 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 06.08.2024. Verfasst von Xu Liu und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Dermatology, West China Hospital, Sichuan University, Chengdu, China. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 7 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Recent advancements in artificial intelligence (AI) and large language models (LLMs) have shown potential in medical fields, including dermatology. With the introduction of image analysis capabilities in LLMs, their application in dermatological diagnostics has garnered significant interest. These capabilities are enabled by the integration of computer vision techniques into the underlying architecture of LLMs.

Objective

This study aimed to compare the diagnostic performance of Claude 3 Opus and ChatGPT with GPT-4 in analyzing dermoscopic images for melanoma detection, providing insights into their strengths and limitations.

Methods

We randomly selected 100 histopathology-confirmed dermoscopic images (50 malignant, 50 benign) from the International Skin Imaging Collaboration (ISIC) archive using a computer-generated randomization process. The ISIC archive was chosen due to its comprehensive and well-annotated collection of dermoscopic images, ensuring a diverse and representative sample. Images were included if they were dermoscopic images of melanocytic lesions with histopathologically confirmed diagnoses. Each model was given the same prompt, instructing it to provide the top 3 differential diagnoses for each image, ranked by likelihood. Primary diagnosis accuracy, accuracy of the top 3 differential diagnoses, and malignancy discrimination ability were assessed. The McNemar test was chosen to compare the diagnostic performance of the 2 models, as it is suitable for analyzing paired nominal data.

Results

In the primary diagnosis, Claude 3 Opus achieved 54.9% sensitivity (95% CI 44.08%-65.37%), 57.14% specificity (95% CI 46.31%-67.46%), and 56% accuracy (95% CI 46.22%-65.42%), while ChatGPT demonstrated 56.86% sensitivity (95% CI 45.99%-67.21%), 38.78% specificity (95% CI 28.77%-49.59%), and 48% accuracy (95% CI 38.37%-57.75%). The McNemar test showed no significant difference between the 2 models (P=.17). For the top 3 differential diagnoses, Claude 3 Opus and ChatGPT included the correct diagnosis in 76% (95% CI 66.33%-83.77%) and 78% (95% CI 68.46%-85.45%) of cases, respectively. The McNemar test showed no significant difference (P=.56). In malignancy discrimination, Claude 3 Opus outperformed ChatGPT with 47.06% sensitivity, 81.63% specificity, and 64% accuracy, compared to 45.1%, 42.86%, and 44%, respectively. The McNemar test showed a significant difference (P

Schlagwörter

Stichwörter der Autor:innen

AIChatGPTClaudeLLMartificial intelligencedermatologistlarge language model

Autor:innen und Institutionen

  • Department of Dermatology, West China Hospital, Sichuan University, Chengdu, China.

ORCID-Profile: Xu Liu, Chaoli Duan, Min-Kyu Kim, Lu Zhang, Eunjin Jee, Beenu Maharjan, Yuwei Huang, Dan Du, Xian Jiang

Zitieren

APA

Liu, X., Duan, C., Kim, M. K., Zhang, L., Jee, E., Maharjan, B., Huang, Y., Du, D., & Jiang, X. (2024). Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. JMIR Medical Informatics, 12, e59273. https://doi.org/10.2196/59273

Vancouver

Liu X, Duan C, Kim MK, Zhang L, Jee E, Maharjan B, et al. Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. JMIR Medical Informatics. 2024;12:e59273. doi:10.2196/59273

BibTeX
@article{Liu2024,
  author = {Liu, Xu and Duan, Chaoli and Kim, Min-Kyu and Zhang, Lu and Jee, Eunjin and Maharjan, Beenu and Huang, Yuwei and Du, Dan and Jiang, Xian},
  title = {Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e59273},
  doi = {10.2196/59273}
}
Kostenlos starten →