Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification
Journal of Engineering Research and Sciences · 2026-09-03 · 5 (9) · 1-14
DOI: 10.55708/js0509001 ↗At a glance
Journal article in Journal of Engineering Research and Sciences, published online on 2026-09-03. Written by Pratheswaran Hariharan and 2 co-authors. 18 references.
Abstract (original language)
Multimodal large language models (MLLMs) can generate fluent visual responses even when the underlying visual prediction is weak, ambiguous, or incorrect. This work presents a retrieval-augmented, reliability-aware selective inference method that evaluates the strength and consistency of visual evidence before a prediction is communicated through a downstream multimodal response. A pretrained ResNet-50 encoder extracts normalized visual embeddings, and FAISS retrieves the top𝑘 reference images from an ImageNet-100 evidence database. Prediction reliability is assessed using retrieval similarity, class-support agreement, evidence margin, entropy-based uncertainty, and an aggregate reliability score. A decision gate then determines whether the prediction should be accepted, presented cautiously, or rejected through abstention or fallback. The selected decision is used to control the final user-facing response generated by the model. Experiments on ImageNet-100 show that the proposed method improves the accuracy of returned predictions from 85.84% to 88.88% at 89.04% coverage. The accepted error rate decreases from 14.16% to 11.12%, corresponding to a 3.04-percentage-point absolute reduction and a 21.48% relative reduction. Expected Calibration Error decreases from 7.40% to 5.87%, while high-reliability wrong predictions decrease from 263 to 226. These results demonstrate the potential of retrieval-derived evidence and selective decision gating as a post-hoc reliability mechanism for controlling visual predictions before they are incorporated into multimodal responses. The current evaluation is conducted in a controlled visual-classification setting and does not constitute a complete assessment of free-form MLLM hallucination.
Cite
APA
Hariharan, P., Xu, H., & Yan, D. (2026). Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification. Journal of Engineering Research and Sciences, 5(9), 1-14. https://doi.org/10.55708/js0509001
Vancouver
Hariharan P, Xu H, Yan D. Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification. Journal of Engineering Research and Sciences. 2026;5(9):1-14. doi:10.55708/js0509001
BibTeX
@article{Hariharan2026,
author = {Hariharan, Pratheswaran and Xu, Haiping and Yan, Donghui},
title = {Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification},
journal = {Journal of Engineering Research and Sciences},
year = {2026},
volume = {5},
number = {9},
pages = {1--14},
doi = {10.55708/js0509001}
}
EUVIMED