EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The simulation of judgment in LLMs

Edoardo Loru iD, Jacopo Nudo iD, Niccolò Di Marco iD, Alessandro Santirocchi iD, Roberto Atzeni iD, Matteo Cinelli iD, Vincenzo Cestari, Clelia Rossi-Arnaud iD, Walter Quattrociocchi iD

Proceedings of the National Academy of Sciences · 21.10.2025 · 122 (42)

DOI: 10.1073/pnas.2518443122 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 13.10.2025. Verfasst von Edoardo Loru und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome 00185, Italy. Hauptthemen laut MEDLINE: Judgment, Language. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) are increasingly embedded in evaluative processes, from information filtering to assessing and addressing knowledge gaps through explanation and credibility judgments. This raises the need to examine how such evaluations are built, what assumptions they rely on, and how their strategies diverge from those of humans. We benchmark six LLMs against expert ratings—NewsGuard and Media Bias/Fact Check—and against human judgments collected through a controlled experiment. We use news domains purely as a controlled benchmark for evaluative tasks, focusing on the underlying mechanisms rather than on news classification per se. To enable direct comparison, we implement a structured agentic framework in which both models and nonexpert participants follow the same evaluation procedure: selecting criteria, retrieving content, and producing justifications. Despite output alignment, our findings show consistent differences in the observable criteria guiding model evaluations, suggesting that lexical associations and statistical priors could influence evaluations in ways that differ from contextual reasoning. This reliance is associated with systematic effects: political asymmetries and a tendency to confuse linguistic form with epistemic reliability—a dynamic we term epistemia, the illusion of knowledge that emerges when surface plausibility replaces verification. Indeed, delegating judgment to such systems may affect the heuristics underlying evaluative processes, suggesting a shift from normative reasoning toward pattern-based approximation and raising open questions about the role of LLMs in evaluative processes.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

JudgmentLanguageHumansMaleFemale

Stichwörter der Autor:innen

Large Language Modelsepistemiaepistemic alignmentevaluation and judgmenthuman–LLM comparison

Autor:innen und Institutionen

  • Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome 00185, Italy.
  • Department of Computer Science, Sapienza University of Rome, Rome 00161, Italy.
  • Department of Legal, Social, and Educational Sciences, Tuscia University, Viterbo 01100, Italy.
  • Department of Psychology, Sapienza University of Rome, Rome 00185, Italy.

ORCID-Profile: Edoardo Loru, Jacopo Nudo, Niccolò Di Marco, Alessandro Santirocchi, Roberto Atzeni, Matteo Cinelli, Clelia Rossi-Arnaud, Walter Quattrociocchi

Zitieren

APA

Loru, E., Nudo, J., Di Marco, N., Santirocchi, A., Atzeni, R., Cinelli, M., Cestari, V., Rossi-Arnaud, C., & Quattrociocchi, W. (2025). The simulation of judgment in LLMs. Proceedings of the National Academy of Sciences, 122(42). https://doi.org/10.1073/pnas.2518443122

Vancouver

Loru E, Nudo J, Di Marco N, Santirocchi A, Atzeni R, Cinelli M, et al. The simulation of judgment in LLMs. Proceedings of the National Academy of Sciences. 2025;122(42). doi:10.1073/pnas.2518443122

BibTeX
@article{Loru2025,
  author = {Loru, Edoardo and Nudo, Jacopo and Di Marco, Niccolò and Santirocchi, Alessandro and Atzeni, Roberto and Cinelli, Matteo and Cestari, Vincenzo and Rossi-Arnaud, Clelia and Quattrociocchi, Walter},
  title = {The simulation of judgment in LLMs},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2025},
  volume = {122},
  number = {42},
  doi = {10.1073/pnas.2518443122}
}
Kostenlos starten →