EUVIMEDOntdek EUVIMED
Open access

Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis

Nicole M Duggan iD, Mike Jin iD, Maria Alejandra Duran Mendicuti iD, Stephen Hallisey iD, Denie Bernier iD, Lauren A Selame iD, Ameneh Asgari-Targhi iD, Chanel E Fischetti iD, Ruben Lucassen iD, Anthony E Samir iD, Erik Duhaime iD, Tina Kapur iD and 1 more

Journal of Medical Internet Research · 2024

DOI: 10.2196/51397 ↗

At a glance

Journal article in Journal of Medical Internet Research, 2024. Written by Nicole M Duggan and 12 co-authors. Institution: Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States. Major MEDLINE topics: Crowdsourcing, Ultrasonography, Lung. Cited 11 times so far (Crossref); 43 references. The full text is freely available in PubMed Central. Funded by: NIBIB NIH HHS.

SamenvattingSubjectsBibliografische gegevensCiteRelated studies

Abstract (original language)

Background

Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality.

Objective

This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data.

Methods

In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips.

Results

Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts’ opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P

Subjects

MeSH terms (MEDLINE)

CrowdsourcingUltrasonographyLungHumansProspective StudiesFemaleMaleMachine LearningAdultMiddle AgedRetrospective Studies

Author keywords

B-linesPOCUSartificial intelligenceclassificationcrowdsourcecrowdsourcedcrowdsourcingdata sciencediagnosediagnosisdiagnosticgamificationgamifiedgamifyimaging

Authors and institutions

  • Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.
  • Centaur Labs, Boston, MA, United States.
  • Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.
  • Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands.
  • Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, United States.
  • Department of Emergency Medicine, Lahey Hospital, University of Massachusetts Medical School, Burlington, MA, United States.

ORCID profiles: Nicole M Duggan, Mike Jin, Maria Alejandra Duran Mendicuti, Stephen Hallisey, Denie Bernier, Lauren A Selame, Ameneh Asgari-Targhi, Chanel E Fischetti, Ruben Lucassen, Anthony E Samir, Erik Duhaime, Tina Kapur, Andrew J Goldsmith

Funding

  • NIBIB NIH HHS

Cite

APA

Duggan, N. M., Jin, M., Duran Mendicuti, M. A., Hallisey, S., Bernier, D., Selame, L. A., Asgari-Targhi, A., Fischetti, C. E., Lucassen, R., Samir, A. E., Duhaime, E., Kapur, T., & Goldsmith, A. J. (2024). Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/51397

Vancouver

Duggan NM, Jin M, Duran Mendicuti MA, Hallisey S, Bernier D, Selame LA, et al. Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis. Journal of Medical Internet Research. 2024. doi:10.2196/51397

BibTeX
@article{Duggan2024,
  author = {Duggan, Nicole M and Jin, Mike and Duran Mendicuti, Maria Alejandra and Hallisey, Stephen and Bernier, Denie and Selame, Lauren A and Asgari-Targhi, Ameneh and Fischetti, Chanel E and Lucassen, Ruben and Samir, Anthony E and Duhaime, Erik and Kapur, Tina and Goldsmith, Andrew J},
  title = {Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/51397}
}
Start for free →