Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0
Journal of Technology Informatics and Engineering · 25.08.2025 · 4 (2) · 502-520
DOI: 10.51903/jtie.v4i2.536 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Technology Informatics and Engineering, online erschienen am 25.08.2025. Verfasst von Ziliang Samuel Zhong und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).
Abstract (Originalsprache)
This paper presents a controlled and reproducible empirical study of evidence-calibrated retrieval-augmented question answering (RAG) for answerable and unanswerable reading-comprehension tasks using the SQuAD 2.0 benchmark. The study focuses on whether a system should abstain when retrieved evidence is insufficient rather than always producing an answer. Six lightweight architectures were evaluated on the full validation set of 11,873 questions, including closed-book, BM25, dense, hybrid, reranked, and a proposed evidence-calibrated hybrid RAG model. The proposed approach combines hybrid top-25 retrieval, lexical reranking, deterministic extractive answering, and evidence sufficiency calibration trained on 43,482 examples. On the validation set, it achieved 31.65% exact match, 34.74% F1, 53.01% answerability accuracy, 53.71% refusal F1, and a 37.49% hallucination-proxy rate. Although overall QA performance remains modest, calibrated evidence sufficiency substantially reduced unsupported answers compared with a forced-answer hybrid reranker, lowering the hallucination-proxy rate from 77.80% while improving F1. However, evidence calibration itself remained weak (AUROC 0.5475, ECE 0.1144). The findings demonstrate that retrieval coverage alone is insufficient to prevent hallucinations and highlight the need for stronger evidence calibration in trustworthy RAG systems.
Zitieren
APA
Zhong, Z. S., Chen, J., Zhong, E., & Sun, X. (2025). Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0. Journal of Technology Informatics and Engineering, 4(2), 502-520. https://doi.org/10.51903/jtie.v4i2.536
Vancouver
Zhong ZS, Chen J, Zhong E, Sun X. Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0. Journal of Technology Informatics and Engineering. 2025;4(2):502-520. doi:10.51903/jtie.v4i2.536
BibTeX
@article{Zhong2025,
author = {Zhong, Ziliang Samuel and Chen, Jing and Zhong, Eric and Sun, Xinzhuo},
title = {Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0},
journal = {Journal of Technology Informatics and Engineering},
year = {2025},
volume = {4},
number = {2},
pages = {502--520},
doi = {10.51903/jtie.v4i2.536}
}
EUVIMED