Evaluating performance of large language models on fundamental structural knowledge
Proceedings of the Institution of Civil Engineers - Structures and Buildings · 25.11.2025 · 178 (11) · 1009-1017
DOI: 10.1680/jstbu.24.00139 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the Institution of Civil Engineers - Structures and Buildings, online erschienen am 21.10.2025. Verfasst von Yi Zhang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 33 Literaturangaben.
Abstract (Originalsprache)
This paper evaluates the performance of multimodal large language models (MLLMs), ChatGPT-4o and Gemini in structural engineering with the image-input approach for the first time in the literature. The study uses 90 real visual-based fundamental mechanics questions that span five subtopics to examine the accuracy of ChatGPT and Gemini and compare them to the response of students. The results show that the students perform better than two MLLMs, although the performance of ChatGPT-4o is closely aligned with student performance or exceeds their performance on topics of kinematics and kinetics, whereas Gemini has the worst performance. However, the study also reveals the limitations of LLMs in accurately extracting mathematical and mechanical information from structural images, as well as their inadequacy in calculations and ensuring logical relationships. Evaluating the capabilities of MLLMs shows immense potential for structural engineering applications, while fine-tuning for specific tasks and developing artificial intelligence agents with MLLMs as the core represent critical directions for future research.
Zitieren
APA
Zhang, Y., & Su, R. K. L. (2025). Evaluating performance of large language models on fundamental structural knowledge. Proceedings of the Institution of Civil Engineers - Structures and Buildings, 178(11), 1009-1017. https://doi.org/10.1680/jstbu.24.00139
Vancouver
Zhang Y, Su RKL. Evaluating performance of large language models on fundamental structural knowledge. Proceedings of the Institution of Civil Engineers - Structures and Buildings. 2025;178(11):1009-1017. doi:10.1680/jstbu.24.00139
BibTeX
@article{Zhang2025,
author = {Zhang, Yi and Su, Ray Kai Leung},
title = {Evaluating performance of large language models on fundamental structural knowledge},
journal = {Proceedings of the Institution of Civil Engineers - Structures and Buildings},
year = {2025},
volume = {178},
number = {11},
pages = {1009--1017},
doi = {10.1680/jstbu.24.00139}
}
EUVIMED