A qualitative assessment of using ChatGPT as large language model for scientific workflow development
GigaScience · 02.01.2024 · 13
DOI: 10.1093/gigascience/giae030 ↗Kurz gefasst
Fachartikel in der Zeitschrift GigaScience, online erschienen am 19.06.2024. Verfasst von Mario Sänger und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Humboldt-Universität zu Berlin, 10099 Berlin, Germany. Hauptthemen laut MEDLINE: Workflow. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 116 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Research Foundation.
Abstract (Originalsprache)
Background
Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages.
Results
To address these challenges, we investigate the efficiency of large language models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed 3 user studies in 2 scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.
Conclusions
Our results show a high accuracy for comprehending and explaining scientific workflows while achieving a reduced performance for modifying and extending workflow descriptions. These findings clearly illustrate the need for further research in this area.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Humboldt-Universität zu Berlin, 10099 Berlin, Germany.
- Department of Geography, Humboldt-Universität zu Berlin, 10099 Berlin, Germany.
- Department of Forest and Wildlife Ecology, University of Wisconsin-Madison, Madison, WI 53706, USA.
ORCID-Profile: Mario Sänger, Ninon De Mecquenem, Katarzyna Ewa Lewińska, Vasilis Bountris, Fabian Lehmann, Ulf Leser, Thomas Kosch
Förderung
- German Research Foundation
Zitieren
APA
Sänger, M., De Mecquenem, N., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024). A qualitative assessment of using ChatGPT as large language model for scientific workflow development. GigaScience, 13. https://doi.org/10.1093/gigascience/giae030
Vancouver
Sänger M, De Mecquenem N, Lewińska KE, Bountris V, Lehmann F, Leser U, et al. A qualitative assessment of using ChatGPT as large language model for scientific workflow development. GigaScience. 2024;13. doi:10.1093/gigascience/giae030
BibTeX
@article{Snger2024,
author = {Sänger, Mario and De Mecquenem, Ninon and Lewińska, Katarzyna Ewa and Bountris, Vasilis and Lehmann, Fabian and Leser, Ulf and Kosch, Thomas},
title = {A qualitative assessment of using ChatGPT as large language model for scientific workflow development},
journal = {GigaScience},
year = {2024},
volume = {13},
doi = {10.1093/gigascience/giae030}
}
EUVIMED