Invalid Is Not Failure: Infrastructure Censoring Bias in Long-Horizon Agent Evaluation
Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 988-992
DOI: 10.1609/aaaiss.v10i1.43517 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Shilin Chhabra.
Abstract (Originalsprache)
Long-running agents sometimes stop without a valid task outcome because a provider, runtime, or harness failed rather than the agent. Benchmarks then score such runs as failures or drop them. We treat this as a missing-data problem. When infrastructure failures are independent of the agent, both policies are closed-form functions of three numbers: the success rate and the mean survival probabilities of successful and failed runs. Scoring invalid runs as failures multiplies the success probability by the survival of successes, so it never overstates reliability yet can reverse model comparisons. Dropping them multiplies the success odds by the survival ratio, so it overstates reliability when successes survive better and understates it otherwise. Rerunning removes only the between-task part of this distortion, whereas a Kaplan–Meier-weighted estimator that uses only observed failure times recovers reliability, even when hazards vary by step. On 80,036 SWE-agent and 4,950 τ-bench trajectories, a hypothetical 1% per-step hazard shifts corpus reliability by up to 6.8 points, and a 0.32% hazard already reverses a τ-bench model pair. The SWE-agent corpus also labels 3,176 runs ended by container runtime errors as task failures, so its reliability is identified only within [16.7%, 20.7%]. We give a rule for separating censoring from failure and a minimum validity record for benchmark reports.
Zitieren
APA
Chhabra, S. (2026). Invalid Is Not Failure: Infrastructure Censoring Bias in Long-Horizon Agent Evaluation. Proceedings of the AAAI Symposium Series, 10(1), 988-992. https://doi.org/10.1609/aaaiss.v10i1.43517
Vancouver
Chhabra S. Invalid Is Not Failure: Infrastructure Censoring Bias in Long-Horizon Agent Evaluation. Proceedings of the AAAI Symposium Series. 2026;10(1):988-992. doi:10.1609/aaaiss.v10i1.43517
BibTeX
@article{Chhabra2026,
author = {Chhabra, Shilin},
title = {Invalid Is Not Failure: Infrastructure Censoring Bias in Long-Horizon Agent Evaluation},
journal = {Proceedings of the AAAI Symposium Series},
year = {2026},
volume = {10},
number = {1},
pages = {988--992},
doi = {10.1609/aaaiss.v10i1.43517}
}
EUVIMED