EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A qualitative assessment of using ChatGPT as large language model for scientific workflow development

Mario Sänger iD, Ninon De Mecquenem iD, Katarzyna Ewa Lewińska iD, Vasilis Bountris iD, Fabian Lehmann iD, Ulf Leser iD, Thomas Kosch iD

GigaScience · 02.01.2024 · 13

DOI: 10.1093/gigascience/giae030 ↗

Kurz gefasst

Fachartikel in der Zeitschrift GigaScience, online erschienen am 19.06.2024. Verfasst von Mario Sänger und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Humboldt-Universität zu Berlin, 10099 Berlin, Germany. Hauptthemen laut MEDLINE: Workflow. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 116 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Research Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages.

Results

To address these challenges, we investigate the efficiency of large language models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed 3 user studies in 2 scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.

Conclusions

Our results show a high accuracy for comprehending and explaining scientific workflows while achieving a reduced performance for modifying and extending workflow descriptions. These findings clearly illustrate the need for further research in this area.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

WorkflowProgramming LanguagesSoftwareComputational BiologyHumans

Stichwörter der Autor:innen

ChatGPTlarge language modelsscientific workflowsuser support

Autor:innen und Institutionen

  • Department of Computer Science, Humboldt-Universität zu Berlin, 10099 Berlin, Germany.
  • Department of Geography, Humboldt-Universität zu Berlin, 10099 Berlin, Germany.
  • Department of Forest and Wildlife Ecology, University of Wisconsin-Madison, Madison, WI 53706, USA.

ORCID-Profile: Mario Sänger, Ninon De Mecquenem, Katarzyna Ewa Lewińska, Vasilis Bountris, Fabian Lehmann, Ulf Leser, Thomas Kosch

Förderung

  • German Research Foundation

Zitieren

APA

Sänger, M., De Mecquenem, N., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024). A qualitative assessment of using ChatGPT as large language model for scientific workflow development. GigaScience, 13. https://doi.org/10.1093/gigascience/giae030

Vancouver

Sänger M, De Mecquenem N, Lewińska KE, Bountris V, Lehmann F, Leser U, et al. A qualitative assessment of using ChatGPT as large language model for scientific workflow development. GigaScience. 2024;13. doi:10.1093/gigascience/giae030

BibTeX
@article{Snger2024,
  author = {Sänger, Mario and De Mecquenem, Ninon and Lewińska, Katarzyna Ewa and Bountris, Vasilis and Lehmann, Fabian and Leser, Ulf and Kosch, Thomas},
  title = {A qualitative assessment of using ChatGPT as large language model for scientific workflow development},
  journal = {GigaScience},
  year = {2024},
  volume = {13},
  doi = {10.1093/gigascience/giae030}
}
Kostenlos starten →