EUVIMEDOpdag EUVIMED
Crossref

Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification

Pratheswaran Hariharan, Haiping Xu, Donghui Yan

Journal of Engineering Research and Sciences · 2026-09-03 · 5 (9) · 1-14

DOI: 10.55708/js0509001 ↗

At a glance

Journal article in Journal of Engineering Research and Sciences, published online on 2026-09-03. Written by Pratheswaran Hariharan and 2 co-authors. 18 references.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Multimodal large language models (MLLMs) can generate fluent visual responses even when the underlying visual prediction is weak, ambiguous, or incorrect. This work presents a retrieval-augmented, reliability-aware selective inference method that evaluates the strength and consistency of visual evidence before a prediction is communicated through a downstream multimodal response. A pretrained ResNet-50 encoder extracts normalized visual embeddings, and FAISS retrieves the top𝑘 reference images from an ImageNet-100 evidence database. Prediction reliability is assessed using retrieval similarity, class-support agreement, evidence margin, entropy-based uncertainty, and an aggregate reliability score. A decision gate then determines whether the prediction should be accepted, presented cautiously, or rejected through abstention or fallback. The selected decision is used to control the final user-facing response generated by the model. Experiments on ImageNet-100 show that the proposed method improves the accuracy of returned predictions from 85.84% to 88.88% at 89.04% coverage. The accepted error rate decreases from 14.16% to 11.12%, corresponding to a 3.04-percentage-point absolute reduction and a 21.48% relative reduction. Expected Calibration Error decreases from 7.40% to 5.87%, while high-reliability wrong predictions decrease from 263 to 226. These results demonstrate the potential of retrieval-derived evidence and selective decision gating as a post-hoc reliability mechanism for controlling visual predictions before they are incorporated into multimodal responses. The current evaluation is conducted in a controlled visual-classification setting and does not constitute a complete assessment of free-form MLLM hallucination.

Cite

APA

Hariharan, P., Xu, H., & Yan, D. (2026). Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification. Journal of Engineering Research and Sciences, 5(9), 1-14. https://doi.org/10.55708/js0509001

Vancouver

Hariharan P, Xu H, Yan D. Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification. Journal of Engineering Research and Sciences. 2026;5(9):1-14. doi:10.55708/js0509001

BibTeX
@article{Hariharan2026,
  author = {Hariharan, Pratheswaran and Xu, Haiping and Yan, Donghui},
  title = {Retrieval-Augmented Reliability-Aware Selective Inference for Visual Classification},
  journal = {Journal of Engineering Research and Sciences},
  year = {2026},
  volume = {5},
  number = {9},
  pages = {1--14},
  doi = {10.55708/js0509001}
}
Start for free →