Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds
Stout in Agriculture and Animal Science · 20.01.2026 · 2 (1) · 24-43
DOI: 10.61424/xbzn8d61 ↗Kurz gefasst
Fachartikel in der Zeitschrift Stout in Agriculture and Animal Science, online erschienen am 20.01.2026. Verfasst von Valerie Zhang.
Abstract (Originalsprache)
Reliable crop-science question answering requires more than a ranking of model accuracies: it requires evidence about paired errors, benchmark artifacts, and the attainable benefit of escalation. This study reanalyzed the 5,045-question CROP benchmark and its archived item-level correctness traces for GPT-3.5, GPT-4 Turbo, Claude-3 Opus, and Qwen-max. Question content and model outcomes were analyzed as separate evidence streams because the benchmark records and outcome workbook do not share a verified item identifier. The content audit found a strong option-length cue: gold options averaged 174.38 characters, compared with 115.38 for distractors, and an earliest-longest deterministic rule answered 4,186 questions correctly (82.97%). In the paired trace, accuracy was 32.86% for GPT-3.5, 85.69% for GPT-4 Turbo, 90.01% for Claude-3 Opus, and 86.64% for Qwen-max. Cochran’s Q showed substantial heterogeneity across the four systems, and all six exact McNemar comparisons remained significant after Holm correction. Claude-3 Opus corrected 331 of GPT-4 Turbo’s 722 errors and 302 of Qwen-max’s 674 errors, yet shared failures limited pairwise any-correct ceilings to 92.25% and 92.63%, respectively. A post-hoc oracle reached those ceilings with small referral budgets, whereas random allocation produced only gradual gains. The findings separate measured complementarity from implementable routing: binary correctness traces support paired accuracy, error overlap, rescue rates, and oracle bounds, but not confidence calibration, reasoning-process attribution, or monetary cost. The study provides an empirically grounded reliability audit and identifies the additional logging required for risk-calibrated agricultural decision support.
Zitieren
APA
Zhang, V. (2026). Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds. Stout in Agriculture and Animal Science, 2(1), 24-43. https://doi.org/10.61424/xbzn8d61
Vancouver
Zhang V. Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds. Stout in Agriculture and Animal Science. 2026;2(1):24-43. doi:10.61424/xbzn8d61
BibTeX
@article{Zhang2026,
author = {Zhang, Valerie},
title = {Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds},
journal = {Stout in Agriculture and Animal Science},
year = {2026},
volume = {2},
number = {1},
pages = {24--43},
doi = {10.61424/xbzn8d61}
}
EUVIMED