How well do AI systems solve AP physics? A comparative evaluation of large language models on algebra-based free response questions
Physics Education · 01.08.2026 · 61 (4) · 045008
DOI: 10.1088/1361-6552/ae700a ↗Kurz gefasst
Fachartikel in der Zeitschrift Physics Education, online erschienen am 17.06.2026. Verfasst von Bilas Paul und 3 weiteren Autor:innen. Die Arbeit nennt 15 Literaturangaben.
Abstract (Originalsprache)
The rapid advancement of large language models has generated growing interest in their potential role in physics education and assessment, yet a focused evaluation of their performance on multi-faceted, free-response physics problems remains underexplored. In this study, we systematically evaluate the performance of four widely accessible AI systems—ChatGPT 4.1 mini, Gemini 2.5 Flash, Claude 4.0 Sonnet, and DeepSeek R1—on algebra-based advanced placement (AP) Physics 1 and AP Physics 2 free-response questions administered between 2015 and 2025. Model-generated solutions were produced under standardized exam-style prompting and evaluated by three independent physics experts using official College Board scoring guidelines. All models achieved relatively high mean scores (82%–92%), indicating strong capability in structured algebraic problem solving. However, substantial year-to-year variability was observed, particularly for AP Physics 1, where statistical testing revealed no consistent performance hierarchy among models. In contrast, AP Physics 2 results showed statistically significant differences, with Gemini and DeepSeek demonstrating more consistent performance than Claude, and numerically higher performance than ChatGPT, though the latter comparisons did not reach statistical significance. A qualitative analysis revealed recurring error patterns across all models, including misinterpretation of diagrams and graphs, incorrect graph construction, incorrect reasoning about vector direction, circuit topology errors, partial and misleading qualitative explanations, and difficulties applying three-dimensional concepts such as the right-hand rule. These findings suggest that while contemporary AI systems can effectively support routine physics problem solving, they remain limited in tasks requiring spatial reasoning, visual interpretation, and conceptual integration. The results highlight both the instructional potential and current pedagogical limitations of AI-assisted learning tools in physics education.
Zitieren
APA
Paul, B., Kaur, J., Chakraborty, S., & Shrestha, S. (2026). How well do AI systems solve AP physics? A comparative evaluation of large language models on algebra-based free response questions. Physics Education, 61(4), 045008. https://doi.org/10.1088/1361-6552/ae700a
Vancouver
Paul B, Kaur J, Chakraborty S, Shrestha S. How well do AI systems solve AP physics? A comparative evaluation of large language models on algebra-based free response questions. Physics Education. 2026;61(4):045008. doi:10.1088/1361-6552/ae700a
BibTeX
@article{Paul2026,
author = {Paul, Bilas and Kaur, Jashandeep and Chakraborty, Shantanu and Shrestha, Shruti},
title = {How well do AI systems solve AP physics? A comparative evaluation of large language models on algebra-based free response questions},
journal = {Physics Education},
year = {2026},
volume = {61},
number = {4},
pages = {045008},
doi = {10.1088/1361-6552/ae700a}
}
EUVIMED