EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0

Ziliang Samuel Zhong, Jing Chen, Eric Zhong, Xinzhuo Sun

Journal of Technology Informatics and Engineering · 25.08.2025 · 4 (2) · 502-520

DOI: 10.51903/jtie.v4i2.536 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Technology Informatics and Engineering, online erschienen am 25.08.2025. Verfasst von Ziliang Samuel Zhong und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents a controlled and reproducible empirical study of evidence-calibrated retrieval-augmented question answering (RAG) for answerable and unanswerable reading-comprehension tasks using the SQuAD 2.0 benchmark. The study focuses on whether a system should abstain when retrieved evidence is insufficient rather than always producing an answer. Six lightweight architectures were evaluated on the full validation set of 11,873 questions, including closed-book, BM25, dense, hybrid, reranked, and a proposed evidence-calibrated hybrid RAG model. The proposed approach combines hybrid top-25 retrieval, lexical reranking, deterministic extractive answering, and evidence sufficiency calibration trained on 43,482 examples. On the validation set, it achieved 31.65% exact match, 34.74% F1, 53.01% answerability accuracy, 53.71% refusal F1, and a 37.49% hallucination-proxy rate. Although overall QA performance remains modest, calibrated evidence sufficiency substantially reduced unsupported answers compared with a forced-answer hybrid reranker, lowering the hallucination-proxy rate from 77.80% while improving F1. However, evidence calibration itself remained weak (AUROC 0.5475, ECE 0.1144). The findings demonstrate that retrieval coverage alone is insufficient to prevent hallucinations and highlight the need for stronger evidence calibration in trustworthy RAG systems.

Zitieren

APA

Zhong, Z. S., Chen, J., Zhong, E., & Sun, X. (2025). Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0. Journal of Technology Informatics and Engineering, 4(2), 502-520. https://doi.org/10.51903/jtie.v4i2.536

Vancouver

Zhong ZS, Chen J, Zhong E, Sun X. Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0. Journal of Technology Informatics and Engineering. 2025;4(2):502-520. doi:10.51903/jtie.v4i2.536

BibTeX
@article{Zhong2025,
  author = {Zhong, Ziliang Samuel and Chen, Jing and Zhong, Eric and Sun, Xinzhuo},
  title = {Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0},
  journal = {Journal of Technology Informatics and Engineering},
  year = {2025},
  volume = {4},
  number = {2},
  pages = {502--520},
  doi = {10.51903/jtie.v4i2.536}
}
Kostenlos starten →