EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education

Marijela Miličević, Mia Rovis, Ratomir Karlović, Sandi Baressi Šegota, Vedran Mrzljak, Ivan Lorencin, Darko Etinger

Information · 21.07.2026 · 17 (7) · 706

DOI: 10.3390/info17070706 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 21.07.2026. Verfasst von Marijela Miličević und 6 weiteren Autor:innen. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: EC Digital Europe Programme EDIH Adria 2.0, SPIN, NextGenerationEU University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly integrated into educational settings, yet their pedagogical reliability remains insufficiently understood. Beyond overt hallucinations, which informed users readily recognize, a subtler failure mode consists of explanations that are coherent, authoritative, and pedagogically plausible while harbouring hidden conceptual flaws, responses we term Socratic traps. This paper introduces SocraticTrap-CS, a publicly available benchmark that probes the capacity of open-weight LLMs to generate such strategic misconceptions on demand. A single structured prompt explicitly elicited three outputs per concept (a correct explanation, an overt hallucination, and a strategic misconception), yielding 735 expert-annotated response segments from seven open-weight models across 35 core concepts in algorithms and data structures, programming languages and paradigms, databases, computer networks, and operating systems. Three domain experts independently annotated each segment using a three-class schema, achieving near-perfect agreement (Fleiss’ κ=0.9487). Because models were explicitly instructed to produce the misconception, the central metric quantifies adversarial instruction-following capacity rather than the base rate of such errors in naturalistic use and should be read as a conservative upper bound on model capability. Under these conditions, compliance reached 91.7% overall (100% for three models; 57.1% for the smallest model, Mistral 7B, whose lower rate plausibly reflects weaker instruction-following rather than greater safety). Expert-judged persuasiveness was moderate to high, errors were predominantly conceptual rather than factual, models differed significantly, and no statistically significant domain-level differences were detected. The benchmark reframes the evaluation of educational LLMs around pedagogical trustworthiness rather than factual correctness alone.

Förderung

  • EC Digital Europe Programme EDIH Adria 2.0
  • SPIN
  • NextGenerationEU University

Zitieren

APA

Miličević, M., Rovis, M., Karlović, R., Šegota, S. B., Mrzljak, V., Lorencin, I., & Etinger, D. (2026). The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education. Information, 17(7), 706. https://doi.org/10.3390/info17070706

Vancouver

Miličević M, Rovis M, Karlović R, Šegota SB, Mrzljak V, Lorencin I, et al. The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education. Information. 2026;17(7):706. doi:10.3390/info17070706

BibTeX
@article{Milievi2026,
  author = {Miličević, Marijela and Rovis, Mia and Karlović, Ratomir and Šegota, Sandi Baressi and Mrzljak, Vedran and Lorencin, Ivan and Etinger, Darko},
  title = {The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education},
  journal = {Information},
  year = {2026},
  volume = {17},
  number = {7},
  pages = {706},
  doi = {10.3390/info17070706}
}
Kostenlos starten →