Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging
Journal of Pioneering Medical Sciences · 05.09.2025 · 14 (S02) · 102-107
DOI: 10.47310/jpms202514s0216 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Pioneering Medical Sciences, online erschienen am 05.09.2025. Verfasst von Madan Kumar und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).
Abstract (Originalsprache)
Background
Early detection of oral potentially malignant disorders (OPMDs) is crucial for improving outcomes in oral cancer, particularly in resource-limited settings. Recent advances in large language models (LLMs) and smartphone imaging provide new opportunities for AI-driven diagnostic support; however, their use in detecting oral mucosal lesions remains underexplored.
Objective
To evaluate and compare the diagnostic performance of Few-shot prompting, Retrieval-Augmented Generation (RAG), and RAG with Chain-of-Thought (RAG + COT) models inthe binary classification of smartphone-captured intraoral buccal mucosa images as either normal or abnormal.
Methods
Using a standardized smartphone protocol, 250 intraoral images from 125 patients were categorized as normal, variations, or lesions and split equally into training and testing sets. Few-shot prompting used a subset only 10 test images — 5 normal and 5 lesion — which may produce unstable estimates, while RAG and RAG + COT models trained on the full training set. Expert annotations guided COT descriptors. Variation images were used only in model training for RAG and RAG+COT to improve contextual representation but excluded from binary performance evaluation. Performance was evaluated via sensitivity, specificity, accuracy, F1 score, precision, recall, and 95% confidence intervals.
Results
Few-shot prompting achieved 80% sensitivity, 100% specificity, 90% accuracy, and an F1 score of 0.88 with wide CIs due to the very small test set. The RAG model, with 54% sensitivity and 91% specificity, showed limited true positive detection. Adding chain-of-thought (RAG + COT) improved sensitivity to 90% and accuracy to 82% (F1: 0.86), though specificity dropped to 64% leading to a higher false-positive rate with potential implications for screening follow-up, however, highlighting the value of structured logical reasoning in enhancing lesion detection.
Conclusion
The RAG + COT model outperformed Few-shot and RAG models in mucosal lesion detection, demonstrating high sensitivity and improved diagnostic accuracy. However, its low specificity highlights the need for human review before acting on AI results. Findings are promising but preliminary, requiring validation in larger and more balanced datasets before clinical adoption. Its structured logical reasoning highlights the potential of LLMs with COT prompting to strengthen community-based oral cancer screening.
Zitieren
APA
Kumar, M., Rajeshwari, S., Savitha, S., Lavanya, C., Ranganathan, K., & Balasubramaniam, A. (2025). Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging. Journal of Pioneering Medical Sciences, 14(S02), 102-107. https://doi.org/10.47310/jpms202514s0216
Vancouver
Kumar M, Rajeshwari S, Savitha S, Lavanya C, Ranganathan K, Balasubramaniam A. Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging. Journal of Pioneering Medical Sciences. 2025;14(S02):102-107. doi:10.47310/jpms202514s0216
BibTeX
@article{Kumar2025,
author = {Kumar, Madan and Rajeshwari, S. and Savitha, S. and Lavanya, C. and Ranganathan, K. and Balasubramaniam, Arthi},
title = {Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging},
journal = {Journal of Pioneering Medical Sciences},
year = {2025},
volume = {14},
number = {S02},
pages = {102--107},
doi = {10.47310/jpms202514s0216}
}
EUVIMED