Artificial Intelligence Explanations of Organic Reaction Mechanisms Revisited: Improved Accuracy and Sophistication but Persistent Mechanistic Flaws
Journal of Chemical Education · 05.11.2026
DOI: 10.1021/acs.jchemed.6c01119 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Chemical Education, online erschienen am 05.11.2026. Verfasst von Charishma Kalicheti und 1 weiteren Autor:innen. Die Arbeit nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Generative artificial intelligence tools are increasingly used by students to explain chemistry concepts, but their educational value depends on the chemical accuracy and mechanistic quality of the explanations. Prior work found that ChatGPT could produce sophisticated and convincing organic reaction mechanisms that were frequently inaccurate. Using the same mechanism set and prompt-cueing levels as those in an earlier study, this study evaluates how a reasoning-based large language model changes the relationships among response correctness, explanation sophistication, and chemical concept use. ChatGPT o3-mini responses were generated for mechanisms spanning a year-long organic chemistry curriculum and analyzed for response-level accuracy, nucleophile and electrophile explanation sophistication, sentence-level concept use, and qualitative reasoning patterns. Compared with GPT-3.5, o3-mini increased the response-level accuracy from 28% to 79% and produced more sophisticated explanations. However, inaccuracies remained concentrated in aromatic ring and carbonyl mechanisms. Prompt cueing increased the chemical richness and explanation sophistication without improving response-level accuracy. Qualitative analyses showed that sophisticated terminology could mask flawed reasoning: o3-mini applied correct chemical concepts to inappropriate mechanisms, overgeneralized common mechanism patterns, and used advanced terminology in vague or decorative ways. These findings show that improvements in aggregate accuracy can coexist with persistent flaws in mechanistic reasoning and that great explanatory detail is not a reliable indicator of validity. The fixed benchmark provides a controlled comparison of which dimensions of response quality changed across model generations, while the sentence-level and qualitative analyses identify recurring error patterns that can be examined in future models. This framework gives instructors concrete features around which to structure student critique and provides researchers with a durable approach for studying AI-generated mechanistic explanations beyond point-in-time capability ranking.
Zitieren
APA
Kalicheti, C., & Yik, B. J. (2026). Artificial Intelligence Explanations of Organic Reaction Mechanisms Revisited: Improved Accuracy and Sophistication but Persistent Mechanistic Flaws. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.6c01119
Vancouver
Kalicheti C, Yik BJ. Artificial Intelligence Explanations of Organic Reaction Mechanisms Revisited: Improved Accuracy and Sophistication but Persistent Mechanistic Flaws. Journal of Chemical Education. 2026. doi:10.1021/acs.jchemed.6c01119
BibTeX
@article{Kalicheti2026,
author = {Kalicheti, Charishma and Yik, Brandon J.},
title = {Artificial Intelligence Explanations of Organic Reaction Mechanisms Revisited: Improved Accuracy and Sophistication but Persistent Mechanistic Flaws},
journal = {Journal of Chemical Education},
year = {2026},
doi = {10.1021/acs.jchemed.6c01119}
}
EUVIMED