EUVIMEDOdkrijte EUVIMED
Open access

Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study

Stephen Bonett iD, Willey Lin iD, Patrina Sexton Topper iD, James Wolfe iD, Jesse Golinkoff iD, Aayushi Deshpande iD, Antonia Villarruel iD, José Bauermeister iD

JMIR Formative Research · 2024-01-12 · 8 · e47091

DOI: 10.2196/47091 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2024-01-12. Written by Stephen Bonett and 7 co-authors. Institution: School of Nursing, University of Pennsylvania, Philadelphia, PA, United States. Cited 59 times so far (Crossref); 47 references. The full text is freely available in PubMed Central. Funded by: NHLBI NIH HHS.

PovzetekSubjectsBibliografski podatkiCiteRelated studies

Abstract (original language)

Background

Web-based surveys increase access to study participation and improve opportunities to reach diverse populations. However, web-based surveys are vulnerable to data quality threats, including fraudulent entries from automated bots and duplicative submissions. Widely used proprietary tools to identify fraud offer little transparency about the methods used, effectiveness, or representativeness of resulting data sets. Robust, reproducible, and context-specific methods of accurately detecting fraudulent responses are needed to ensure integrity and maximize the value of web-based survey research.

Objective

This study aims to describe a multilayered fraud detection system implemented in a large web-based survey about COVID-19 attitudes, beliefs, and behaviors; examine the agreement between this fraud detection system and a proprietary fraud detection system; and compare the resulting study samples from each of the 2 fraud detection methods.

Methods

The PhillyCEAL Common Survey is a cross-sectional web-based survey that remotely enrolled residents ages 13 years and older to assess how the COVID-19 pandemic impacted individuals, neighborhoods, and communities in Philadelphia, Pennsylvania. Two fraud detection methods are described and compared: (1) a multilayer fraud detection strategy developed by the research team that combined automated validation of response data and real-time verification of study entries by study personnel and (2) the proprietary fraud detection system used by the Qualtrics (Qualtrics) survey platform. Descriptive statistics were computed for the full sample and for responses classified as valid by 2 different fraud detection methods, and classification tables were created to assess agreement between the methods. The impact of fraud detection methods on the distribution of vaccine confidence by racial or ethnic group was assessed.

Results

Of 7950 completed surveys, our multilayer fraud detection system identified 3228 (40.60%) cases as valid, while the Qualtrics fraud detection system identified 4389 (55.21%) cases as valid. The 2 methods showed only “fair” or “minimal” agreement in their classifications (κ=0.25; 95% CI 0.23-0.27). The choice of fraud detection method impacted the distribution of vaccine confidence by racial or ethnic group.

Conclusions

The selection of a fraud detection method can affect the study’s sample composition. The findings of this study, while not conclusive, suggest that a multilayered approach to fraud detection that includes conservative use of automated fraud detection and integration of human review of entries tailored to the study’s specific context and its participants may be warranted for future survey research.

Subjects

Author keywords

COVID-19Philadelphiacross-sectionaldatadata privacydata protectiondata qualitydata validationfraudfraud detectioninformation securityprivacysecuritysurveysurvey methodology

Authors and institutions

  • School of Nursing, University of Pennsylvania, Philadelphia, PA, United States.
  • Department of Psychology, Ashoka University, Sonepat, India.

ORCID profiles: Stephen Bonett, Willey Lin, Patrina Sexton Topper, James Wolfe, Jesse Golinkoff, Aayushi Deshpande, Antonia Villarruel, José Bauermeister

Funding

  • NHLBI NIH HHS

Cite

APA

Bonett, S., Lin, W., Sexton Topper, P., Wolfe, J., Golinkoff, J., Deshpande, A., Villarruel, A., & Bauermeister, J. (2024). Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study. JMIR Formative Research, 8, e47091. https://doi.org/10.2196/47091

Vancouver

Bonett S, Lin W, Sexton Topper P, Wolfe J, Golinkoff J, Deshpande A, et al. Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study. JMIR Formative Research. 2024;8:e47091. doi:10.2196/47091

BibTeX
@article{Bonett2024,
  author = {Bonett, Stephen and Lin, Willey and Sexton Topper, Patrina and Wolfe, James and Golinkoff, Jesse and Deshpande, Aayushi and Villarruel, Antonia and Bauermeister, José},
  title = {Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e47091},
  doi = {10.2196/47091}
}
Start for free →