Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis
Journal of Medical Internet Research · 2024
DOI: 10.2196/51397 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2024. Verfasst von Nicole M Duggan und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States. Hauptthemen laut MEDLINE: Crowdsourcing, Ultrasonography, Lung. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIBIB NIH HHS.
Abstract (Originalsprache)
Background
Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality.
Objective
This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data.
Methods
In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips.
Results
Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts’ opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.
- Centaur Labs, Boston, MA, United States.
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.
- Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands.
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, United States.
- Department of Emergency Medicine, Lahey Hospital, University of Massachusetts Medical School, Burlington, MA, United States.
ORCID-Profile: Nicole M Duggan, Mike Jin, Maria Alejandra Duran Mendicuti, Stephen Hallisey, Denie Bernier, Lauren A Selame, Ameneh Asgari-Targhi, Chanel E Fischetti, Ruben Lucassen, Anthony E Samir, Erik Duhaime, Tina Kapur, Andrew J Goldsmith
Förderung
- NIBIB NIH HHS
Zitieren
APA
Duggan, N. M., Jin, M., Duran Mendicuti, M. A., Hallisey, S., Bernier, D., Selame, L. A., Asgari-Targhi, A., Fischetti, C. E., Lucassen, R., Samir, A. E., Duhaime, E., Kapur, T., & Goldsmith, A. J. (2024). Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/51397
Vancouver
Duggan NM, Jin M, Duran Mendicuti MA, Hallisey S, Bernier D, Selame LA, et al. Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis. Journal of Medical Internet Research. 2024. doi:10.2196/51397
BibTeX
@article{Duggan2024,
author = {Duggan, Nicole M and Jin, Mike and Duran Mendicuti, Maria Alejandra and Hallisey, Stephen and Bernier, Denie and Selame, Lauren A and Asgari-Targhi, Ameneh and Fischetti, Chanel E and Lucassen, Ruben and Samir, Anthony E and Duhaime, Erik and Kapur, Tina and Goldsmith, Andrew J},
title = {Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis},
journal = {Journal of Medical Internet Research},
year = {2024},
doi = {10.2196/51397}
}
EUVIMED