Evaluating AI performance on end-to-end data science projects
PLOS One · 11.11.2026 · 21 (11) · e0361887
DOI: 10.1371/journal.pone.0361887 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 11.11.2026. Verfasst von Evelyn Hughes und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mathematics, University of Toronto, Toronto, Ontario, Canada. Hauptthemen laut MEDLINE: Data Science, Artificial Intelligence. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council of Canada, Data Sciences Institute, University of Toronto, Social Sciences and Humanities Research Council of Canada.
Abstract (Originalsprache)
Generative AI is used for knowledge work but many current AI evaluations focus on isolated tasks and constituent parts such as writing code, rather than end-to-end workflows. We test whether AI models can generate end-to-end data science projects. This requires AI models to select a dataset, develop a research question, write code to download, clean, and test data, create figures, and write a reproducible paper. To do this we collect 40 real end-to-end data science projects, then grade them according to a rubric. We use these to build an automated grading pipeline that evaluates data science projects and aligns with the grades that human graders give. For each of nine AI models we then generate five projects and use the automated grader to evaluate them. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment such as explaining measurement, writing abstracts, and citing sources. Anthropic’s Claude Opus 4.8 averaged 87 per cent and OpenAI’s GPT-5.4 averaged 85 per cent, both around the level of an A for an undergraduate. No model was consistently the best across all aspects of the rubric. These findings suggest that while AI models could soon approximate entry-level data scientists on routine tasks, they require consistent verification and evaluation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Mathematics, University of Toronto, Toronto, Ontario, Canada.
- Faculty of Information and Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada.
ORCID-Profile: Rohan Alexander
Förderung
- Natural Sciences and Engineering Research Council of Canada
- Data Sciences Institute, University of Toronto
- Social Sciences and Humanities Research Council of Canada
Zitieren
APA
Hughes, E., & Alexander, R. (2026). Evaluating AI performance on end-to-end data science projects. PLOS One, 21(11), e0361887. https://doi.org/10.1371/journal.pone.0361887
Vancouver
Hughes E, Alexander R. Evaluating AI performance on end-to-end data science projects. PLOS One. 2026;21(11):e0361887. doi:10.1371/journal.pone.0361887
BibTeX
@article{Hughes2026,
author = {Hughes, Evelyn and Alexander, Rohan},
title = {Evaluating AI performance on end-to-end data science projects},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {11},
pages = {e0361887},
doi = {10.1371/journal.pone.0361887}
}
EUVIMED