EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Calibrating LLM-Derived Trust Scores for News Outlets When Public Factuality Scorecards Disappear

Pieter Claassen, Gary van Vuuren, Tanja Verster

Information · 08.09.2026 · 17 (9) · 867

DOI: 10.3390/info17090867 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 08.09.2026. Verfasst von Pieter Claassen und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Third-party news-source factuality scorecards are valuable but increasingly fragile. Web pages change, access conditions shift and underlying datasets may disappear. The challenge is therefore not only benchmark imperfection but also benchmark sustainability as credibility datasets, search interfaces and platform reputation signals become harder to access reproducibly. This study investigates whether a fixed large language model (LLM) scoring procedure can generate durable, replayable outlet-level trust scores that align with a frozen external factuality benchmark rather than objective ground truth. Fifty-two English-language news outlets were assessed across nine predefined trust dimensions and compared with a frozen Media Bias Fact Check (MBFC) factuality snapshot. Raw LLM scores were rank-aware but compressed (Pearson’s r=0.801, Spearman’s ρ=0.843, full-cohort mean GAP =0.221). An affine calibration fitted on 42 training outlets increased full-cohort Pearson alignment to r=0.828 and reduced mean GAP to 0.090; on the fixed ten-outlet validation fold, mean GAP fell from 0.162 to 0.063. Across 1000 additional stratified 42/10 splits, median validation GAP was 0.078 (central 95% split range 0.048–0.110). Wikipedia lead and source-weighted web enrichment did not outperform the calibrated archival path in the retained data. The Step 4 unweighted web-search meter improved on the Wikipedia-lead meter (Pearson’s r=0.697, Spearman’s ρ=0.576, full-cohort mean GAP =0.200; fixed-validation GAP =0.146) but remained below the calibrated archival path. RSS monitoring is reported separately as an asymmetric, bounded adverse-event signal rather than a second factuality benchmark. These findings support calibrated LLM trust vectors as a potentially useful archival proxy while highlighting benchmark dependence, sampling constraints, model sensitivity and the importance of reproducible data provenance.

Zitieren

APA

Claassen, P., Vuuren, G. V., & Verster, T. (2026). Calibrating LLM-Derived Trust Scores for News Outlets When Public Factuality Scorecards Disappear. Information, 17(9), 867. https://doi.org/10.3390/info17090867

Vancouver

Claassen P, Vuuren GV, Verster T. Calibrating LLM-Derived Trust Scores for News Outlets When Public Factuality Scorecards Disappear. Information. 2026;17(9):867. doi:10.3390/info17090867

BibTeX
@article{Claassen2026,
  author = {Claassen, Pieter and Vuuren, Gary van and Verster, Tanja},
  title = {Calibrating LLM-Derived Trust Scores for News Outlets When Public Factuality Scorecards Disappear},
  journal = {Information},
  year = {2026},
  volume = {17},
  number = {9},
  pages = {867},
  doi = {10.3390/info17090867}
}
Kostenlos starten →