EUVIMEDEUVIMED entdecken
Crossref

Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis

Yash Rajeshbhai Parikh

Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 387-396

DOI: 10.1609/aaaiss.v10i1.43430 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Yash Rajeshbhai Parikh.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

An operator cannot check whether a language model’s diagnosis of an outage is right, but can check its evidence: each claim names an entity, a measurement and a value that the telemetry supports or not. Our checker passes the usual validation, confirming 96.1% of expert-chosen measurements and catching 97.5% of corrupted ones. Yet a blind hand check of 352 claims models actually wrote upheld 86.7% of its confirmations and only 17.6% of its accusations, mostly because of four ordinary bugs in tolerance, number parsing, claim scope and entity lookup. Correcting its count with that sample puts the hallucination rate at 2.8% (95% CI 1.2 to 4.7), not the 13.3% it reported. A checker must itself be checked.

Zitieren

APA

Parikh, Y. R. (2026). Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis. Proceedings of the AAAI Symposium Series, 10(1), 387-396. https://doi.org/10.1609/aaaiss.v10i1.43430

Vancouver

Parikh YR. Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis. Proceedings of the AAAI Symposium Series. 2026;10(1):387-396. doi:10.1609/aaaiss.v10i1.43430

BibTeX
@article{Parikh2026,
  author = {Parikh, Yash Rajeshbhai},
  title = {Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis},
  journal = {Proceedings of the AAAI Symposium Series},
  year = {2026},
  volume = {10},
  number = {1},
  pages = {387--396},
  doi = {10.1609/aaaiss.v10i1.43430}
}
Kostenlos starten →