EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment

Giulia Polverini, Jakob Melin, Elias Önerud, Bor Gregorcic

Physical Review Physics Education Research · 27.05.2025 · 21 (1)

DOI: 10.1103/physrevphyseducres.21.010154 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Physical Review Physics Education Research, online erschienen am 27.05.2025. Verfasst von Giulia Polverini und 3 weiteren Autor:innen. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

[This paper is part of the Focused Collection in Artificial Intelligence Tools in Physics Teaching and Physics Education Research.] Artificial intelligence-based chatbots are increasingly influencing physics education because of their ability to interpret and respond to textual and visual inputs. This study evaluates the performance of two large multimodal model-based chatbots, ChatGPT-4 and ChatGPT-4o, on the brief electricity and magnetism assessment (BEMA), a conceptual physics inventory rich in visual representations such as vector fields, circuit diagrams, and graphs. Quantitative analysis shows that ChatGPT-4o outperforms both ChatGPT-4 and a large sample of university students, and demonstrates improvements in ChatGPT-4o’s vision interpretation ability over its predecessor ChatGPT-4. However, qualitative analysis of ChatGPT-4o’s responses reveals persistent challenges. We identified three types of difficulties in the chatbot’s responses to tasks on BEMA: (i) difficulties with visual interpretation, (ii) difficulties in providing correct physics laws or rules, and (iii) difficulties with spatial coordination and application of physics representations. Spatial reasoning tasks, particularly those requiring the use of the right-hand rule, proved especially problematic. These findings highlight that the most broadly used large multimodal model-based chatbot, ChatGPT-4o, still exhibits significant difficulties in engaging with physics tasks involving visual representations. While the chatbot shows potential for educational applications, including personalized tutoring and accessibility support for students who are blind or have low vision, its limitations necessitate caution. On the other hand, our findings can also be leveraged to design assessments that are difficult for chatbots to solve.

Zitieren

APA

Polverini, G., Melin, J., Önerud, E., & Gregorcic, B. (2025). Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment. Physical Review Physics Education Research, 21(1). https://doi.org/10.1103/physrevphyseducres.21.010154

Vancouver

Polverini G, Melin J, Önerud E, Gregorcic B. Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment. Physical Review Physics Education Research. 2025;21(1). doi:10.1103/physrevphyseducres.21.010154

BibTeX
@article{Polverini2025,
  author = {Polverini, Giulia and Melin, Jakob and Önerud, Elias and Gregorcic, Bor},
  title = {Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment},
  journal = {Physical Review Physics Education Research},
  year = {2025},
  volume = {21},
  number = {1},
  doi = {10.1103/physrevphyseducres.21.010154}
}
Kostenlos starten →