Benchmarking large language models for materials synthesis: The case of atomic layer deposition
Journal of Vacuum Science & Technology A · 01.05.2025 · 43 (3)
DOI: 10.1116/6.0004319 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Vacuum Science & Technology A, online erschienen am 09.04.2025. Verfasst von Angel Yanguas-Gil und 7 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 19 Literaturangaben. Gefördert von: Argonne National Laboratory, Advanced Materials for Energy-Water Systems.
Abstract (Originalsprache)
In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. This emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.
Förderung
- Argonne National Laboratory
- Advanced Materials for Energy-Water Systems
Zitieren
APA
Yanguas-Gil, A., Dearing, M. T., Elam, J. W., Jones, J. C., Kim, S., Mohammad, A., Nguyen, C. T., & Sengupta, B. (2025). Benchmarking large language models for materials synthesis: The case of atomic layer deposition. Journal of Vacuum Science & Technology A, 43(3). https://doi.org/10.1116/6.0004319
Vancouver
Yanguas-Gil A, Dearing MT, Elam JW, Jones JC, Kim S, Mohammad A, et al. Benchmarking large language models for materials synthesis: The case of atomic layer deposition. Journal of Vacuum Science & Technology A. 2025;43(3). doi:10.1116/6.0004319
BibTeX
@article{YanguasGil2025,
author = {Yanguas-Gil, Angel and Dearing, Matthew T. and Elam, Jeffrey W. and Jones, Jessica C. and Kim, Sungjoon and Mohammad, Adnan and Nguyen, Chi Thang and Sengupta, Bratin},
title = {Benchmarking large language models for materials synthesis: The case of atomic layer deposition},
journal = {Journal of Vacuum Science & Technology A},
year = {2025},
volume = {43},
number = {3},
doi = {10.1116/6.0004319}
}
EUVIMED