Capability-Profiled Agricultural Language-Model Systems: Calibration and Cost-Aware Routing across Memorization, Understanding, Reasoning, and Generation
Stout in Agriculture and Animal Science · 20.01.2026 · 2 (1) · 01-23
DOI: 10.61424/g6rbs789 ↗Kurz gefasst
Fachartikel in der Zeitschrift Stout in Agriculture and Animal Science, online erschienen am 20.01.2026. Verfasst von Sylvia He.
Abstract (Originalsprache)
Agricultural language systems must span factual recall, diagnosis, numerical reasoning, and management advice, yet a single score conceals these differences. This study profiles the AgriEval 2025 snapshot using 14,697 multiple-choice and 2,167 open-answer questions across six domains, 29 subdomains, 15 tasks, and four capability families. Duplicate-normalized question groups were assigned to five stratified folds before training, calibration, and held-out testing. Experiments compared Qwen2.5-0.5B-Instruct in direct and self-verification conditions with position, character TF-IDF, case-retrieval, fusion, and answer-copy baselines. On 2,940 multiple-choice test questions, CharPair achieved the highest exact accuracy, 0.3833 (95% confidence interval [0.3653, 0.3997]); Qwen direct reached 0.3286 [0.3119, 0.3459]. Isotonic regression reduced Qwen’s 10-bin expected calibration error from 0.2646 to 0.0147 without changing answers. Self-verification reduced accuracy to 0.2922: 261 errors were corrected, but 368 correct direct answers were corrupted. A calibration-trained CharPair-to-Qwen router sent 9.01% of questions to Qwen, attained 0.3810 accuracy, and required 60.455 ms per question, compared with 0.3833 and 0.549 ms for CharPair alone. On 433 open-answer items, Qwen obtained character ROUGE-L of 0.1555 [0.1477, 0.1636], while VerifiedRAG reached 0.1641. The experiments show that calibration can improve probability reliability even when retrieval, verification, and routing do not improve answer quality. Capability, subgroup, uncertainty, and measured-compute profiles therefore provide a more informative assessment than one leaderboard score.
Zitieren
APA
He, S. (2026). Capability-Profiled Agricultural Language-Model Systems: Calibration and Cost-Aware Routing across Memorization, Understanding, Reasoning, and Generation. Stout in Agriculture and Animal Science, 2(1), 01-23. https://doi.org/10.61424/g6rbs789
Vancouver
He S. Capability-Profiled Agricultural Language-Model Systems: Calibration and Cost-Aware Routing across Memorization, Understanding, Reasoning, and Generation. Stout in Agriculture and Animal Science. 2026;2(1):01-23. doi:10.61424/g6rbs789
BibTeX
@article{He2026,
author = {He, Sylvia},
title = {Capability-Profiled Agricultural Language-Model Systems: Calibration and Cost-Aware Routing across Memorization, Understanding, Reasoning, and Generation},
journal = {Stout in Agriculture and Animal Science},
year = {2026},
volume = {2},
number = {1},
pages = {01--23},
doi = {10.61424/g6rbs789}
}
EUVIMED