A refined approach for evaluating small datasets via binary classification using machine learning
PLOS ONE · 21.05.2024 · 19 (5) · e0301276
DOI: 10.1371/journal.pone.0301276 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS ONE, online erschienen am 21.05.2024. Verfasst von Steffen Steinert und 6 weiteren Autor:innen. Beteiligte Einrichtung: Chair of Physics Education, Ludwig-Maximilians-Universität München (LMU Munich), Munich, Germany. Hauptthemen laut MEDLINE: Machine Learning. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Classical statistical analysis of data can be complemented or replaced with data analysis based on machine learning. However, in certain disciplines, such as education research, studies are frequently limited to small datasets, which raises several questions regarding biases and coincidentally positive results. In this study, we present a refined approach for evaluating the performance of a binary classification based on machine learning for small datasets. The approach includes a non-parametric permutation test as a method to quantify the probability of the results generalising to new data. Furthermore, we found that a repeated nested cross-validation is almost free of biases and yields reliable results that are only slightly dependent on chance. Considering the advantages of several evaluation metrics, we suggest a combination of more than one metric to train and evaluate machine learning classifiers. In the specific case that both classes are equally important, the Matthews correlation coefficient exhibits the lowest bias and chance for coincidentally good results. The results indicate that it is essential to avoid several biases when analysing small datasets using machine learning.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Chair of Physics Education, Ludwig-Maximilians-Universität München (LMU Munich), Munich, Germany.
- Department of Electrical and Computer Engineering, RPTU Kaiserslautern-Landau, Germany.
- Smart Data & Knowledge Services, German Research Center for Artificial Intelligence, Kaiserslautern, Germany.
- Human-Centered Ubiquitous Media, Ludwig-Maximilians-Universität München (LMU Munich), Munich, Germany.
ORCID-Profile: Steffen Steinert, Verena Ruf, Jochen Kuhn, Stefan Küchemann
Zitieren
APA
Steinert, S., Ruf, V., Dzsotjan, D., Großmann, N., Schmidt, A., Kuhn, J., & Küchemann, S. (2024). A refined approach for evaluating small datasets via binary classification using machine learning. PLOS ONE, 19(5), e0301276. https://doi.org/10.1371/journal.pone.0301276
Vancouver
Steinert S, Ruf V, Dzsotjan D, Großmann N, Schmidt A, Kuhn J, et al. A refined approach for evaluating small datasets via binary classification using machine learning. PLOS ONE. 2024;19(5):e0301276. doi:10.1371/journal.pone.0301276
BibTeX
@article{Steinert2024,
author = {Steinert, Steffen and Ruf, Verena and Dzsotjan, David and Großmann, Nicolas and Schmidt, Albrecht and Kuhn, Jochen and Küchemann, Stefan},
title = {A refined approach for evaluating small datasets via binary classification using machine learning},
journal = {PLOS ONE},
year = {2024},
volume = {19},
number = {5},
pages = {e0301276},
doi = {10.1371/journal.pone.0301276}
}
EUVIMED