Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images
Endocrine · 11.10.2024 · 87 (3) · 1041-1049
DOI: 10.1007/s12020-024-04066-x ↗Kurz gefasst
Fachartikel in der Zeitschrift Endocrine, online erschienen am 11.10.2024. Verfasst von Ziman Chen und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China. chenzm27@mail3.sysu.edu.cn. Hauptthemen laut MEDLINE: Thyroid Nodule. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hong Kong Polytechnic University.
Abstract (Originalsprache)
Purpose
Large language models (LLMs) are pivotal in artificial intelligence, demonstrating advanced capabilities in natural language understanding and multimodal interactions, with significant potential in medical applications. This study explores the feasibility and efficacy of LLMs, specifically ChatGPT-4o and Claude 3-Opus, in classifying thyroid nodules using ultrasound images.
Methods
This study included 112 patients with a total of 116 thyroid nodules, comprising 75 benign and 41 malignant cases. Ultrasound images of these nodules were analyzed using ChatGPT-4o and Claude 3-Opus to diagnose the benign or malignant nature of the nodules. An independent evaluation by a junior radiologist was also conducted. Diagnostic performance was assessed using Cohen’s Kappa and receiver operating characteristic (ROC) curve analysis, referencing pathological diagnoses.
Results
ChatGPT-4o demonstrated poor agreement with pathological results ( Kappa = 0.116), while Claude 3-Opus showed even lower agreement ( Kappa = 0.034). The junior radiologist exhibited moderate agreement ( Kappa = 0.450). ChatGPT-4o achieved an area under the ROC curve (AUC) of 57.0% (95% CI: 48.6–65.5%), slightly outperforming Claude 3-Opus (AUC of 52.0%, 95% CI: 43.2–60.9%). In contrast, the junior radiologist achieved a significantly higher AUC of 72.4% (95% CI: 63.7–81.1%). The unnecessary biopsy rates were 41.4% for ChatGPT-4o, 43.1% for Claude 3-Opus, and 12.1% for the junior radiologist.
Conclusion
While LLMs such as ChatGPT-4o and Claude 3-Opus show promise for future applications in medical imaging, their current use in clinical diagnostics should be approached cautiously due to their limited accuracy.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China. chenzm27@mail3.sysu.edu.cn.
- School of Healthcare Sciences, Cardiff University, Cardiff, UK.
- Department of Ultrasound, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China.
- Department of Surgery, North District Hospital, Sheung Shui, New Territories, Hong Kong, China.
- Department of Surgery, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong, China.
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China.
Förderung
- Hong Kong Polytechnic University
Zitieren
APA
Chen, Z., Chambara, N., Wu, C., Lo, X., Liu, S. Y. W., Gunda, S. T., Han, X., Qu, J., Chen, F., & Ying, M. T. C. (2024). Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images. Endocrine, 87(3), 1041-1049. https://doi.org/10.1007/s12020-024-04066-x
Vancouver
Chen Z, Chambara N, Wu C, Lo X, Liu SYW, Gunda ST, et al. Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images. Endocrine. 2024;87(3):1041-1049. doi:10.1007/s12020-024-04066-x
BibTeX
@article{Chen2024,
author = {Chen, Ziman and Chambara, Nonhlanhla and Wu, Chaoqun and Lo, Xina and Liu, Shirley Yuk Wah and Gunda, Simon Takadiyi and Han, Xinyang and Qu, Jingguo and Chen, Fei and Ying, Michael Tin Cheung},
title = {Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images},
journal = {Endocrine},
year = {2024},
volume = {87},
number = {3},
pages = {1041--1049},
doi = {10.1007/s12020-024-04066-x}
}
EUVIMED