EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias

Ajay Bhasin, Sue Bielinski, Abel N Kho, Nicholas Larson, Laura J Rasmussen-Torvik

ACI Open · 01.07.2024 · 08 (02) · e89-e93

DOI: 10.1055/a-2402-5937 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACI Open, erschienen am 01.07.2024. Verfasst von Ajay Bhasin und 4 weiteren Autor:innen. Beteiligte Einrichtung: Northwestern University Feinberg School of Medicine, Chicago, United States. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NHGRI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Computable or electronic phenotypes of patient conditions are becoming more commonplace in quality improvement and clinical research. During phenotyping algorithm validation, standard classification performance measures (i.e., sensitivity, specificity, positive predictive value, negative predictive value, and accuracy) are often employed. When validation is performed on a randomly sampled patient population, direct estimates of these measures are valid. However, studies will commonly sample patients conditional on the algorithm result prior to validation, leading to a form of bias known as verification bias.

Methods

We illustrate validation study sampling design and naïve and bias-corrected validation performance through both a concrete example (1,000 cases, 100 noncases, 1:1 sampling on predicted status) and a more thorough simulation study under varied realistic scenarios. We additionally describe the development of a free web calculator to adjust estimates for people validating phenotyping algorithms.

Results

In our illustrative example, naïve performance estimates corresponded to 0.942 sensitivity, 0.979 specificity, and 0.960 accuracy; these contrast proper estimates of 0.620 sensitivity, 0.999 specificity, and 0.944 accuracy after adjusting for verification bias using our free calculator. Our simulation results demonstrate increasing positive bias for sensitivity and negative bias for specificity as the disease prevalence approaches zero, with decreasing positive predictive value moderately exacerbating these biases.

Conclusion

Novel computable phenotypes of patient conditions must account for verification bias when calculating performance measures of the algorithm. The performance measures may vary significantly based on disease prevalence in the source population so use of a free web calculator to adjust these measures is desirable.

Schlagwörter

Stichwörter der Autor:innen

data analysisdata collectionevaluationstatistical methodssystem improvement

Autor:innen und Institutionen

  • Northwestern University Feinberg School of Medicine, Chicago, United States.
  • Mayo Clinic Alix School of Medicine, Rochester, Minnesota, United States.

Förderung

  • NHGRI NIH HHS

Zitieren

APA

Bhasin, A., Bielinski, S., Kho, A. N., Larson, N., & Rasmussen-Torvik, L. J. (2024). An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias. ACI Open, 08(02), e89-e93. https://doi.org/10.1055/a-2402-5937

Vancouver

Bhasin A, Bielinski S, Kho AN, Larson N, Rasmussen-Torvik LJ. An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias. ACI Open. 2024;08(02):e89-e93. doi:10.1055/a-2402-5937

BibTeX
@article{Bhasin2024,
  author = {Bhasin, Ajay and Bielinski, Sue and Kho, Abel N and Larson, Nicholas and Rasmussen-Torvik, Laura J},
  title = {An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias},
  journal = {ACI Open},
  year = {2024},
  volume = {08},
  number = {02},
  pages = {e89--e93},
  doi = {10.1055/a-2402-5937}
}
Kostenlos starten →