EUVIMEDEUVIMED entdecken
Crossref

Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds

Valerie Zhang

Stout in Agriculture and Animal Science · 20.01.2026 · 2 (1) · 24-43

DOI: 10.61424/xbzn8d61 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Stout in Agriculture and Animal Science, online erschienen am 20.01.2026. Verfasst von Valerie Zhang.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reliable crop-science question answering requires more than a ranking of model accuracies: it requires evidence about paired errors, benchmark artifacts, and the attainable benefit of escalation. This study reanalyzed the 5,045-question CROP benchmark and its archived item-level correctness traces for GPT-3.5, GPT-4 Turbo, Claude-3 Opus, and Qwen-max. Question content and model outcomes were analyzed as separate evidence streams because the benchmark records and outcome workbook do not share a verified item identifier. The content audit found a strong option-length cue: gold options averaged 174.38 characters, compared with 115.38 for distractors, and an earliest-longest deterministic rule answered 4,186 questions correctly (82.97%). In the paired trace, accuracy was 32.86% for GPT-3.5, 85.69% for GPT-4 Turbo, 90.01% for Claude-3 Opus, and 86.64% for Qwen-max. Cochran’s Q showed substantial heterogeneity across the four systems, and all six exact McNemar comparisons remained significant after Holm correction. Claude-3 Opus corrected 331 of GPT-4 Turbo’s 722 errors and 302 of Qwen-max’s 674 errors, yet shared failures limited pairwise any-correct ceilings to 92.25% and 92.63%, respectively. A post-hoc oracle reached those ceilings with small referral budgets, whereas random allocation produced only gradual gains. The findings separate measured complementarity from implementable routing: binary correctness traces support paired accuracy, error overlap, rescue rates, and oracle bounds, but not confidence calibration, reasoning-process attribution, or monetary cost. The study provides an empirically grounded reliability audit and identifies the additional logging required for risk-calibrated agricultural decision support.

Zitieren

APA

Zhang, V. (2026). Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds. Stout in Agriculture and Animal Science, 2(1), 24-43. https://doi.org/10.61424/xbzn8d61

Vancouver

Zhang V. Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds. Stout in Agriculture and Animal Science. 2026;2(1):24-43. doi:10.61424/xbzn8d61

BibTeX
@article{Zhang2026,
  author = {Zhang, Valerie},
  title = {Correctness-Trace Replay of Commercial LLMs on CROP: Paired Performance, Shortcut Risk, and Oracle Referral Bounds},
  journal = {Stout in Agriculture and Animal Science},
  year = {2026},
  volume = {2},
  number = {1},
  pages = {24--43},
  doi = {10.61424/xbzn8d61}
}
Kostenlos starten →