Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis
Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 387-396
DOI: 10.1609/aaaiss.v10i1.43430 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Yash Rajeshbhai Parikh.
Abstract (Originalsprache)
An operator cannot check whether a language model’s diagnosis of an outage is right, but can check its evidence: each claim names an entity, a measurement and a value that the telemetry supports or not. Our checker passes the usual validation, confirming 96.1% of expert-chosen measurements and catching 97.5% of corrupted ones. Yet a blind hand check of 352 claims models actually wrote upheld 86.7% of its confirmations and only 17.6% of its accusations, mostly because of four ordinary bugs in tolerance, number parsing, claim scope and entity lookup. Correcting its count with that sample puts the hallucination rate at 2.8% (95% CI 1.2 to 4.7), not the 13.3% it reported. A checker must itself be checked.
Zitieren
APA
Parikh, Y. R. (2026). Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis. Proceedings of the AAAI Symposium Series, 10(1), 387-396. https://doi.org/10.1609/aaaiss.v10i1.43430
Vancouver
Parikh YR. Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis. Proceedings of the AAAI Symposium Series. 2026;10(1):387-396. doi:10.1609/aaaiss.v10i1.43430
BibTeX
@article{Parikh2026,
author = {Parikh, Yash Rajeshbhai},
title = {Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis},
journal = {Proceedings of the AAAI Symposium Series},
year = {2026},
volume = {10},
number = {1},
pages = {387--396},
doi = {10.1609/aaaiss.v10i1.43430}
}
EUVIMED