EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Comparative analysis of AI and expert evaluations in engineering design pedagogy

Tuğra Karademir Coşkun iD, Esra Bozkurt Altan

PLOS One · 22.09.2025 · 20 (9) · e0332715

DOI: 10.1371/journal.pone.0332715 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift PLOS One, online erschienen am 22.09.2025. Verfasst von Tuğra Karademir Coşkun und 1 weiteren Autor:innen. Beteiligte Einrichtung: Computer Education, Faculty of Education, Sinop, Turkiye. Hauptthemen laut MEDLINE: Engineering, Artificial Intelligence. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 124 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Türkiye Bilimsel ve Teknolojik AraÅŸtırma Kurumu.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Integrating engineering design processes into science education has become a significant priority in STEM instruction. However, many science teachers face difficulties incorporating these processes due to limited pedagogical expertise. Generative artificial intelligence (GAI) tools such as ChatGPT offer potential support mechanisms by evaluating lesson plans and providing formative feedback. This study investigates the reliability and validity of GAI evaluations compared to expert assessments.

Methods

This mixed-methods study involved 43 science teachers who received professional development over four months to integrate engineering design into their lesson plans. A total of 52 lesson plans were evaluated using structured and unstructured prompts via ChatGPT 4.5, alongside evaluations by expert mentors. Quantitative data were analyzed using the Intraclass Correlation Coefficient (ICC) and Bland-Altman methods to assess inter-rater consistency. Qualitative data was analyzed through open and deductive coding to interpret differences in evaluation rationale.

Results

Findings revealed high consistency between structured prompt AI evaluations and expert assessments (ICC = 0.708), while unstructured prompts showed low and non-significant agreement (ICC = 0.076). Qualitative analysis indicated that AI evaluations, particularly those using structured prompts, tend to be more positive and holistic, whereas experts offered more detailed and critical feedback. Differences were also observed in evaluating dcomponents like problem definition, testability, and interdisciplinary integration.

Conclusion

Structured AI prompts offer reliable and valid evaluation results comparable to expert assessments and could serve as scalable tools in teacher support systems. However, unstructured prompts produce inconsistent outcomes and require refinement. The study highlights both the potential and limitations of using GAI tools for pedagogical evaluation in STEM education.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EngineeringArtificial IntelligenceHumansTeachingFemaleMaleReproducibility of Results

Autor:innen und Institutionen

  • Computer Education, Faculty of Education, Sinop, Turkiye.
  • Science Education, Faculty of Education, Sinop, Turkiye.

ORCID-Profile: Tuğra Karademir Coşkun

Förderung

  • Türkiye Bilimsel ve Teknolojik AraÅŸtırma Kurumu

Zitieren

APA

Coşkun, T. K., & Altan, E. B. (2025). Comparative analysis of AI and expert evaluations in engineering design pedagogy. PLOS One, 20(9), e0332715. https://doi.org/10.1371/journal.pone.0332715

Vancouver

Coşkun TK, Altan EB. Comparative analysis of AI and expert evaluations in engineering design pedagogy. PLOS One. 2025;20(9):e0332715. doi:10.1371/journal.pone.0332715

BibTeX
@article{Cokun2025,
  author = {Coşkun, Tuğra Karademir and Altan, Esra Bozkurt},
  title = {Comparative analysis of AI and expert evaluations in engineering design pedagogy},
  journal = {PLOS One},
  year = {2025},
  volume = {20},
  number = {9},
  pages = {e0332715},
  doi = {10.1371/journal.pone.0332715}
}
Kostenlos starten →