Calibration Quality and Query-Level Risk Detection in BM25 Retrieval
Future Internet · 19.11.2026 · 18 (11) · 639
DOI: 10.3390/fi18110639 ↗Kurz gefasst
Fachartikel in der Zeitschrift Future Internet, online erschienen am 19.11.2026. Verfasst von Khaled Albishre. Die Arbeit nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Umm Al-Qura University, Saudi Arabia.
Abstract (Originalsprache)
Internet-facing search and retrieval-augmented generation systems often use retrieval scores as confidence signals to route, filter, or withhold results, although ranking quality and probabilistic confidence are different properties. We test whether better-calibrated BM25 scores are more useful for such query-level decisions, using MS MARCO passages with TREC Deep Learning 2019 and 2020 judgments. Naive min–max normalization, Platt scaling, and isotonic regression are compared in a two-fold cross-collection design, and detection of below-median retrieval is evaluated with a fold-stratified area under the receiver operating characteristic curve (AUROC). Naive normalization was not calibrated across collections, and calibrating raw scores did not outperform a constant predictor at the training-set prevalence (pooled Brier score 0.211 for both). Normalizing before calibration gave the lowest Brier scores (0.201–0.202) and the lowest miscalibration in point estimate. The estimators with lower point-estimate calibration error did not demonstrate better risk detection: they reached AUROCs of 0.562–0.568, close to normalization alone (0.562) and below a top-100 score-dispersion baseline (0.731). This held for top-100 calibrated aggregates, absolute effectiveness thresholds, and calibrators refitted in the bootstrap, although moderate gains or losses cannot be excluded. Calibration quality and decision utility should be validated separately before retrieval scores serve as control signals.
Förderung
- Umm Al-Qura University, Saudi Arabia
Zitieren
APA
Albishre, K. (2026). Calibration Quality and Query-Level Risk Detection in BM25 Retrieval. Future Internet, 18(11), 639. https://doi.org/10.3390/fi18110639
Vancouver
Albishre K. Calibration Quality and Query-Level Risk Detection in BM25 Retrieval. Future Internet. 2026;18(11):639. doi:10.3390/fi18110639
BibTeX
@article{Albishre2026,
author = {Albishre, Khaled},
title = {Calibration Quality and Query-Level Risk Detection in BM25 Retrieval},
journal = {Future Internet},
year = {2026},
volume = {18},
number = {11},
pages = {639},
doi = {10.3390/fi18110639}
}
EUVIMED