Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations
Journal of Business Logistics · 2026-07 · 47 (3)
DOI: 10.1111/jbl.70072 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Business Logistics, online erschienen am 14.05.2026. Verfasst von Finnegan A. McKinley und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This study compares Generative Artificial Intelligence (GAI) to human procurement professionals on supplier evaluation tasks. Using Structural Topic Modeling (STM) on 123 government supplier bids from 31 projects solicited by the State of Ohio between January 2023 and December 2024, we compare evaluations from three reasoning models (o3, Grok‐3‐Mini, DeepSeek R1‐0528) against human evaluators. Adopting a signaling theory perspective, we find asymmetry in signal processing between GAI and human evaluators. GAI demonstrates high consistency and strong human alignment when evaluating compliance signals (e.g., technical specifications), which makes it suitable for qualification screening. However, GAI exhibits high scoring volatility with competitive signals (e.g., value‐add propositions), indicating that human judgment remains critical for assessing differentiation. We also find that the number of bidders influences signal composition, with compliance signals more prevalent in less competitive solicitations. The findings suggest a two‐stage evaluation framework where GAI handles compliance screening and humans focus on competitive assessment. GAI scoring volatility serves as a canary‐in‐the‐mine to identify when human oversight is necessary.
Zitieren
APA
McKinley, F. A., Dohmen, A. E., & Castillo, V. E. (2026). Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations. Journal of Business Logistics, 47(3). https://doi.org/10.1111/jbl.70072
Vancouver
McKinley FA, Dohmen AE, Castillo VE. Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations. Journal of Business Logistics. 2026;47(3). doi:10.1111/jbl.70072
BibTeX
@article{McKinley2026,
author = {McKinley, Finnegan A. and Dohmen, Anne E. and Castillo, Vincent E.},
title = {Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations},
journal = {Journal of Business Logistics},
year = {2026},
volume = {47},
number = {3},
doi = {10.1111/jbl.70072}
}
EUVIMED