Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging
Journal of Pioneering Medical Sciences · 2025-09-05 · 14 (S02) · 102-107
DOI: 10.47310/jpms202514s0216 ↗At a glance
Journal article in Journal of Pioneering Medical Sciences, published online on 2025-09-05. Written by Madan Kumar and 5 co-authors. Cited 3 times so far (Crossref).
Abstract (original language)
Background
Early detection of oral potentially malignant disorders (OPMDs) is crucial for improving outcomes in oral cancer, particularly in resource-limited settings. Recent advances in large language models (LLMs) and smartphone imaging provide new opportunities for AI-driven diagnostic support; however, their use in detecting oral mucosal lesions remains underexplored.
Objective
To evaluate and compare the diagnostic performance of Few-shot prompting, Retrieval-Augmented Generation (RAG), and RAG with Chain-of-Thought (RAG + COT) models inthe binary classification of smartphone-captured intraoral buccal mucosa images as either normal or abnormal.
Methods
Using a standardized smartphone protocol, 250 intraoral images from 125 patients were categorized as normal, variations, or lesions and split equally into training and testing sets. Few-shot prompting used a subset only 10 test images — 5 normal and 5 lesion — which may produce unstable estimates, while RAG and RAG + COT models trained on the full training set. Expert annotations guided COT descriptors. Variation images were used only in model training for RAG and RAG+COT to improve contextual representation but excluded from binary performance evaluation. Performance was evaluated via sensitivity, specificity, accuracy, F1 score, precision, recall, and 95% confidence intervals.
Results
Few-shot prompting achieved 80% sensitivity, 100% specificity, 90% accuracy, and an F1 score of 0.88 with wide CIs due to the very small test set. The RAG model, with 54% sensitivity and 91% specificity, showed limited true positive detection. Adding chain-of-thought (RAG + COT) improved sensitivity to 90% and accuracy to 82% (F1: 0.86), though specificity dropped to 64% leading to a higher false-positive rate with potential implications for screening follow-up, however, highlighting the value of structured logical reasoning in enhancing lesion detection.
Conclusion
The RAG + COT model outperformed Few-shot and RAG models in mucosal lesion detection, demonstrating high sensitivity and improved diagnostic accuracy. However, its low specificity highlights the need for human review before acting on AI results. Findings are promising but preliminary, requiring validation in larger and more balanced datasets before clinical adoption. Its structured logical reasoning highlights the potential of LLMs with COT prompting to strengthen community-based oral cancer screening.
Cite
APA
Kumar, M., Rajeshwari, S., Savitha, S., Lavanya, C., Ranganathan, K., & Balasubramaniam, A. (2025). Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging. Journal of Pioneering Medical Sciences, 14(S02), 102-107. https://doi.org/10.47310/jpms202514s0216
Vancouver
Kumar M, Rajeshwari S, Savitha S, Lavanya C, Ranganathan K, Balasubramaniam A. Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging. Journal of Pioneering Medical Sciences. 2025;14(S02):102-107. doi:10.47310/jpms202514s0216
BibTeX
@article{Kumar2025,
author = {Kumar, Madan and Rajeshwari, S. and Savitha, S. and Lavanya, C. and Ranganathan, K. and Balasubramaniam, Arthi},
title = {Performance Evaluation of Large Language Models in Detecting Buccal Mucosal Lesions Using Smartphone-Based Imaging},
journal = {Journal of Pioneering Medical Sciences},
year = {2025},
volume = {14},
number = {S02},
pages = {102--107},
doi = {10.47310/jpms202514s0216}
}
EUVIMED