EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows

Naomi Esposito, Ali Aghazadeh Ardebili, Massimo Stella

Computers · 18.11.2026 · 15 (11) · 849

DOI: 10.3390/computers15110849 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 18.11.2026. Verfasst von Naomi Esposito und 2 weiteren Autor:innen. Die Arbeit nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministero dell’Università e della Ricerca, CALCOLO, funded by Fondazione VRT, Call for Research COGNOSCO/COGNASCO, funded by the University of Trento.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly used in math education, yet little is known about how persona conditioning reshapes LLM math explanations. We examine 28,000 runs from the Math Education Digital Shadows (MEDS) dataset, spanning 14 LLMs as AI assistants (i.e., vanilla AI) or as human shadows (i.e., personifications). We analyze qualitative mathematics-related responses through textual forma mentis networks, knowledge graphs including word–word syntactic, semantic and emotional associations. We also use a DistilBERT-based classifier to flag five categories of potential logical fallacies. A regex-based check of the PCA responses showed that reassuring language was frequent but model-dependent and was not always flagged as appeal to emotion. Persona conditioning produced model-dependent changes in network size, density, and algebraic connectivity. Aggregated human-shadow responses displayed broader lexical coverage. Answers to easy and medium math questions often had no classifier-flagged fallacies, whereas an anxiety-framed PCA prompt sharply increased appeal-to-emotion and relevance labels, especially in AI-assistant mode. On an 18-item quiz, AI assistants’ accuracy exceeded human shadows’ in 10 out of 14 architectures. Expressed confidence exceeded empirical accuracy in 13 of 14 AI-assistant and 11 of 14 human-shadow conditions, a descriptive gap, not a calibration measure. These results show that (i) cognitive digital shadowing modulates LLM-generated mathematical discourse compared to the vanilla AI assistant case, and (ii) the tested LLMs are not infallible mathematics oracles. No measure here directly assesses the explanations’ mathematical validity or teaching quality.

Förderung

  • Ministero dell’Università e della Ricerca
  • CALCOLO, funded by Fondazione VRT
  • Call for Research COGNOSCO/COGNASCO, funded by the University of Trento

Zitieren

APA

Esposito, N., Ardebili, A. A., & Stella, M. (2026). LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows. Computers, 15(11), 849. https://doi.org/10.3390/computers15110849

Vancouver

Esposito N, Ardebili AA, Stella M. LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows. Computers. 2026;15(11):849. doi:10.3390/computers15110849

BibTeX
@article{Esposito2026,
  author = {Esposito, Naomi and Ardebili, Ali Aghazadeh and Stella, Massimo},
  title = {LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {849},
  doi = {10.3390/computers15110849}
}
Kostenlos starten →