EUVIMEDObjevte EUVIMED
Open access

Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study

Scott Silvey iD, Jinze Liu iD

Journal of Medical Internet Research · 2024-12-17 · 26 · e60231

DOI: 10.2196/60231 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2024-12-17. Written by Scott Silvey and 1 co-authors. Institution: Department of Biostatistics, School of Public Health, Virginia Commonwealth University, Richmond, VA, United States. Major MEDLINE topics: Algorithms. Cited 66 times so far (Crossref); 42 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

The performance of a classification algorithm eventually reaches a point of diminishing returns, where the additional sample added does not improve the results. Thus, there is a need to determine an optimal sample size that maximizes performance while accounting for computational burden or budgetary concerns.

Objective

This study aimed to determine optimal sample sizes and the relationships between sample size and dataset-level characteristics over a variety of binary classification algorithms.

Methods

A total of 16 large open-source datasets were collected, each containing a binary clinical outcome. Furthermore, 4 machine learning algorithms were assessed: XGBoost (XGB), random forest (RF), logistic regression (LR), and neural networks (NNs). For each dataset, the cross-validated area under the curve (AUC) was calculated at increasing sample sizes, and learning curves were fit. Sample sizes needed to reach the observed full–dataset AUC minus 2 points (0.02) were calculated from the fitted learning curves and compared across the datasets and algorithms. Dataset–level characteristics, minority class proportion, full–dataset AUC, number of features, type of features, and degree of nonlinearity were examined. Negative binomial regression models were used to quantify relationships between these characteristics and expected sample sizes within each algorithm. A total of 4 multivariable models were constructed, which selected the best-fitting combination of dataset–level characteristics.

Results

Among the 16 datasets (full-dataset sample sizes ranging from 70,000-1,000,000), median sample sizes were 9960 (XGB), 3404 (RF), 696 (LR), and 12,298 (NN) to reach AUC stability. For all 4 algorithms, more balanced classes (multiplier: 0.93-0.96 for a 1% increase in minority class proportion) were associated with decreased sample size. Other characteristics varied in importance across algorithms—in general, more features, weaker features, and more complex relationships between the predictors and the response increased expected sample sizes. In multivariable analysis, the top selected predictors were minority class proportion among all 4 algorithms assessed, full–dataset AUC (XGB, RF, and NN), and dataset nonlinearity (XGB, RF, and NN). For LR, the top predictors were minority class proportion, percentage of strong linear features, and number of features. Final multivariable sample size models had high goodness-of-fit, with dataset–level predictors explaining a majority (66.5%-84.5%) of the total deviance in the data among all 4 models.

Conclusions

The sample sizes needed to reach AUC stability among 4 popular classification algorithms vary by dataset and method and are associated with dataset–level characteristics that can be influenced or estimated before the start of a research study.

Subjects

MeSH terms (MEDLINE)

AlgorithmsSample SizeHumansMachine Learning

Author keywords

MLalgorithmanalysesanalysisclassification algorithmclinical researchcurve analysisdatasetdecision makingdecision treesguidelineslearning-curve analysismachine learningmedical informaticsresearch design

Authors and institutions

  • Department of Biostatistics, School of Public Health, Virginia Commonwealth University, Richmond, VA, United States.

ORCID profiles: Scott Silvey, Jinze Liu

Cite

APA

Silvey, S., & Liu, J. (2024). Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study. Journal of Medical Internet Research, 26, e60231. https://doi.org/10.2196/60231

Vancouver

Silvey S, Liu J. Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study. Journal of Medical Internet Research. 2024;26:e60231. doi:10.2196/60231

BibTeX
@article{Silvey2024,
  author = {Silvey, Scott and Liu, Jinze},
  title = {Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e60231},
  doi = {10.2196/60231}
}
Start for free →