EUVIMEDSkopri EUVIMED
Open access · CC BY 4.0

LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows

Naomi Esposito, Ali Aghazadeh Ardebili, Massimo Stella

Computers · 2026-11-18 · 15 (11) · 849

DOI: 10.3390/computers15110849 ↗

At a glance

Journal article in Computers, published online on 2026-11-18. Written by Naomi Esposito and 2 co-authors. 50 references. The full text is freely available (open access, CC BY 4.0). Funded by: Ministero dell’Università e della Ricerca, CALCOLO, funded by Fondazione VRT, Call for Research COGNOSCO/COGNASCO, funded by the University of Trento.

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Large language models (LLMs) are increasingly used in math education, yet little is known about how persona conditioning reshapes LLM math explanations. We examine 28,000 runs from the Math Education Digital Shadows (MEDS) dataset, spanning 14 LLMs as AI assistants (i.e., vanilla AI) or as human shadows (i.e., personifications). We analyze qualitative mathematics-related responses through textual forma mentis networks, knowledge graphs including word–word syntactic, semantic and emotional associations. We also use a DistilBERT-based classifier to flag five categories of potential logical fallacies. A regex-based check of the PCA responses showed that reassuring language was frequent but model-dependent and was not always flagged as appeal to emotion. Persona conditioning produced model-dependent changes in network size, density, and algebraic connectivity. Aggregated human-shadow responses displayed broader lexical coverage. Answers to easy and medium math questions often had no classifier-flagged fallacies, whereas an anxiety-framed PCA prompt sharply increased appeal-to-emotion and relevance labels, especially in AI-assistant mode. On an 18-item quiz, AI assistants’ accuracy exceeded human shadows’ in 10 out of 14 architectures. Expressed confidence exceeded empirical accuracy in 13 of 14 AI-assistant and 11 of 14 human-shadow conditions, a descriptive gap, not a calibration measure. These results show that (i) cognitive digital shadowing modulates LLM-generated mathematical discourse compared to the vanilla AI assistant case, and (ii) the tested LLMs are not infallible mathematics oracles. No measure here directly assesses the explanations’ mathematical validity or teaching quality.

Funding

  • Ministero dell’Università e della Ricerca
  • CALCOLO, funded by Fondazione VRT
  • Call for Research COGNOSCO/COGNASCO, funded by the University of Trento

Cite

APA

Esposito, N., Ardebili, A. A., & Stella, M. (2026). LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows. Computers, 15(11), 849. https://doi.org/10.3390/computers15110849

Vancouver

Esposito N, Ardebili AA, Stella M. LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows. Computers. 2026;15(11):849. doi:10.3390/computers15110849

BibTeX
@article{Esposito2026,
  author = {Esposito, Naomi and Ardebili, Ali Aghazadeh and Stella, Massimo},
  title = {LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {849},
  doi = {10.3390/computers15110849}
}
Start for free →