The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education
Information · 21.07.2026 · 17 (7) · 706
DOI: 10.3390/info17070706 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 21.07.2026. Verfasst von Marijela Miličević und 6 weiteren Autor:innen. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: EC Digital Europe Programme EDIH Adria 2.0, SPIN, NextGenerationEU University.
Abstract (Originalsprache)
Large language models (LLMs) are increasingly integrated into educational settings, yet their pedagogical reliability remains insufficiently understood. Beyond overt hallucinations, which informed users readily recognize, a subtler failure mode consists of explanations that are coherent, authoritative, and pedagogically plausible while harbouring hidden conceptual flaws, responses we term Socratic traps. This paper introduces SocraticTrap-CS, a publicly available benchmark that probes the capacity of open-weight LLMs to generate such strategic misconceptions on demand. A single structured prompt explicitly elicited three outputs per concept (a correct explanation, an overt hallucination, and a strategic misconception), yielding 735 expert-annotated response segments from seven open-weight models across 35 core concepts in algorithms and data structures, programming languages and paradigms, databases, computer networks, and operating systems. Three domain experts independently annotated each segment using a three-class schema, achieving near-perfect agreement (Fleiss’ κ=0.9487). Because models were explicitly instructed to produce the misconception, the central metric quantifies adversarial instruction-following capacity rather than the base rate of such errors in naturalistic use and should be read as a conservative upper bound on model capability. Under these conditions, compliance reached 91.7% overall (100% for three models; 57.1% for the smallest model, Mistral 7B, whose lower rate plausibly reflects weaker instruction-following rather than greater safety). Expert-judged persuasiveness was moderate to high, errors were predominantly conceptual rather than factual, models differed significantly, and no statistically significant domain-level differences were detected. The benchmark reframes the evaluation of educational LLMs around pedagogical trustworthiness rather than factual correctness alone.
Förderung
- EC Digital Europe Programme EDIH Adria 2.0
- SPIN
- NextGenerationEU University
Zitieren
APA
Miličević, M., Rovis, M., Karlović, R., Šegota, S. B., Mrzljak, V., Lorencin, I., & Etinger, D. (2026). The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education. Information, 17(7), 706. https://doi.org/10.3390/info17070706
Vancouver
Miličević M, Rovis M, Karlović R, Šegota SB, Mrzljak V, Lorencin I, et al. The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education. Information. 2026;17(7):706. doi:10.3390/info17070706
BibTeX
@article{Milievi2026,
author = {Miličević, Marijela and Rovis, Mia and Karlović, Ratomir and Šegota, Sandi Baressi and Mrzljak, Vedran and Lorencin, Ivan and Etinger, Darko},
title = {The Socratic Trap: Benchmarking the Capacity of Large Language Models to Generate Strategic Misconceptions in Computer Science Education},
journal = {Information},
year = {2026},
volume = {17},
number = {7},
pages = {706},
doi = {10.3390/info17070706}
}
EUVIMED