Can AI see what the surgeons see? Analyzing the accuracy of ChatGPT: publicly available multimodal AI for radiographic interpretation and treatment plan in oral surgery

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Background: With the rise of AI, patients have been actively doing their own research. With emergence of easily accessible multimodal AI, such as ChatGPT-5 that can interpret images, there is a growing need to evaluate their accuracy, reliability, and potential clinical application. Therefore, this study aimed to evaluate the accuracy of radiographic diagnostics and surgical treatment planning of ChatGPT-5 in the context of oral and maxillofacial surgery. Methods: We conducted a cross-sectional study using ChatGPT-5’s free version and departmental OPG. Oral surgeons opinions regarding diagnosis and treatment plan were recorded, and a Cohen’s Kappa was calculated for inter-rater agreeability. Furthermore, Cohen’s Kappa was calculated amongst the Surgeons and AI’s response for inter-rater reliability analysis. Results: The mean number of positive findings per radiograph was significantly higher for ChatGPT (3.69±1.28) compared to oral surgeons (2.8±1.18), with this difference reaching statistical significance (p=0.001). Indicating that AI consistently reported more diagnostic and treatment related observations per OPG. Conclusions: The findings demonstrated that ChatGPT-5 identified a significantly higher number of diagnostic and treatment related findings per radiograph compared to clinicians. While this suggests high sensitivity, qualitative and agreement analysis (Cohen’s Kappa) revealed that many of these additional findings represented false positive interpretations.