EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations

Finnegan A. McKinley, Anne E. Dohmen, Vincent E. Castillo

Journal of Business Logistics · 2026-07 · 47 (3)

DOI: 10.1111/jbl.70072 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Business Logistics, online erschienen am 14.05.2026. Verfasst von Finnegan A. McKinley und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study compares Generative Artificial Intelligence (GAI) to human procurement professionals on supplier evaluation tasks. Using Structural Topic Modeling (STM) on 123 government supplier bids from 31 projects solicited by the State of Ohio between January 2023 and December 2024, we compare evaluations from three reasoning models (o3, Grok‐3‐Mini, DeepSeek R1‐0528) against human evaluators. Adopting a signaling theory perspective, we find asymmetry in signal processing between GAI and human evaluators. GAI demonstrates high consistency and strong human alignment when evaluating compliance signals (e.g., technical specifications), which makes it suitable for qualification screening. However, GAI exhibits high scoring volatility with competitive signals (e.g., value‐add propositions), indicating that human judgment remains critical for assessing differentiation. We also find that the number of bidders influences signal composition, with compliance signals more prevalent in less competitive solicitations. The findings suggest a two‐stage evaluation framework where GAI handles compliance screening and humans focus on competitive assessment. GAI scoring volatility serves as a canary‐in‐the‐mine to identify when human oversight is necessary.

Zitieren

APA

McKinley, F. A., Dohmen, A. E., & Castillo, V. E. (2026). Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations. Journal of Business Logistics, 47(3). https://doi.org/10.1111/jbl.70072

Vancouver

McKinley FA, Dohmen AE, Castillo VE. Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations. Journal of Business Logistics. 2026;47(3). doi:10.1111/jbl.70072

BibTeX
@article{McKinley2026,
  author = {McKinley, Finnegan A. and Dohmen, Anne E. and Castillo, Vincent E.},
  title = {Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations},
  journal = {Journal of Business Logistics},
  year = {2026},
  volume = {47},
  number = {3},
  doi = {10.1111/jbl.70072}
}
Kostenlos starten →