An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias
ACI Open · 01.07.2024 · 08 (02) · e89-e93
DOI: 10.1055/a-2402-5937 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACI Open, erschienen am 01.07.2024. Verfasst von Ajay Bhasin und 4 weiteren Autor:innen. Beteiligte Einrichtung: Northwestern University Feinberg School of Medicine, Chicago, United States. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NHGRI NIH HHS.
Abstract (Originalsprache)
Objectives
Computable or electronic phenotypes of patient conditions are becoming more commonplace in quality improvement and clinical research. During phenotyping algorithm validation, standard classification performance measures (i.e., sensitivity, specificity, positive predictive value, negative predictive value, and accuracy) are often employed. When validation is performed on a randomly sampled patient population, direct estimates of these measures are valid. However, studies will commonly sample patients conditional on the algorithm result prior to validation, leading to a form of bias known as verification bias.
Methods
We illustrate validation study sampling design and naïve and bias-corrected validation performance through both a concrete example (1,000 cases, 100 noncases, 1:1 sampling on predicted status) and a more thorough simulation study under varied realistic scenarios. We additionally describe the development of a free web calculator to adjust estimates for people validating phenotyping algorithms.
Results
In our illustrative example, naïve performance estimates corresponded to 0.942 sensitivity, 0.979 specificity, and 0.960 accuracy; these contrast proper estimates of 0.620 sensitivity, 0.999 specificity, and 0.944 accuracy after adjusting for verification bias using our free calculator. Our simulation results demonstrate increasing positive bias for sensitivity and negative bias for specificity as the disease prevalence approaches zero, with decreasing positive predictive value moderately exacerbating these biases.
Conclusion
Novel computable phenotypes of patient conditions must account for verification bias when calculating performance measures of the algorithm. The performance measures may vary significantly based on disease prevalence in the source population so use of a free web calculator to adjust these measures is desirable.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Northwestern University Feinberg School of Medicine, Chicago, United States.
- Mayo Clinic Alix School of Medicine, Rochester, Minnesota, United States.
Förderung
- NHGRI NIH HHS
Zitieren
APA
Bhasin, A., Bielinski, S., Kho, A. N., Larson, N., & Rasmussen-Torvik, L. J. (2024). An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias. ACI Open, 08(02), e89-e93. https://doi.org/10.1055/a-2402-5937
Vancouver
Bhasin A, Bielinski S, Kho AN, Larson N, Rasmussen-Torvik LJ. An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias. ACI Open. 2024;08(02):e89-e93. doi:10.1055/a-2402-5937
BibTeX
@article{Bhasin2024,
author = {Bhasin, Ajay and Bielinski, Sue and Kho, Abel N and Larson, Nicholas and Rasmussen-Torvik, Laura J},
title = {An Online Tool for Correcting Performance Measures of Electronic Phenotyping Algorithms for Verification Bias},
journal = {ACI Open},
year = {2024},
volume = {08},
number = {02},
pages = {e89--e93},
doi = {10.1055/a-2402-5937}
}
EUVIMED