EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence

Mitchell M Conover iD, Patrick B Ryan, Yong Chen iD, Marc A Suchard iD, George Hripcsak, Martijn J Schuemie

Journal of the American Medical Informatics Association · 2025

DOI: 10.1093/jamia/ocae317 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2025. Verfasst von Mitchell M Conover und 5 weiteren Autor:innen. Beteiligte Einrichtung: Coordinating Center, Observational Health Data Science and Informatics, New York City, NY 10032, United States. Hauptthemen laut MEDLINE: Observational Studies as Topic, Research Design. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NHLBI NIH HHS, NLM NIH HHS, NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Propose a framework to empirically evaluate and report validity of findings from observational studies using pre-specified objective diagnostics, increasing trust in real-world evidence (RWE). Materials and Methods The framework employs objective diagnostic measures to assess the appropriateness of study designs, analytic assumptions, and threats to validity in generating reliable evidence addressing causal questions. Diagnostic evaluations should be interpreted before the unblinding of study results or, alternatively, only unblind results from analyses that pass pre-specified thresholds. We provide a conceptual overview of objective diagnostic measures and demonstrate their impact on the validity of RWE from a large-scale comparative new-user study of various antihypertensive medications. We evaluated expected absolute systematic error (EASE) before and after applying diagnostic thresholds, using a large set of negative control outcomes.

Results

Applying objective diagnostics reduces bias and improves evidence reliability in observational studies. Among 11 716 analyses (EASE = 0.38), 13.9% met pre-specified diagnostic thresholds which reduced EASE to zero. Objective diagnostics provide a comprehensive and empirical set of tests that increase confidence when passed and raise doubts when failed.

Discussion

The increasing use of real-world data presents a scientific opportunity; however, the complexity of the evidence generation process poses challenges for understanding study validity and trusting RWE. Deploying objective diagnostics is crucial to reducing bias and improving reliability in RWE generation. Under ideal conditions, multiple study designs pass diagnostics and generate consistent results, deepening understanding of causal relationships. Open-source, standardized programs can facilitate implementation of diagnostic analyses.

Conclusion

Objective diagnostics are a valuable addition to the RWE generation process.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Observational Studies as TopicResearch DesignHumansReproducibility of ResultsTrustBias

Stichwörter der Autor:innen

causalitydata interpretationmethodsobservational studyresearch designstatistical

Autor:innen und Institutionen

  • Coordinating Center, Observational Health Data Science and Informatics, New York City, NY 10032, United States.
  • Observational Health Data Analytics, Johnson & Johnson, Titusville, NJ 08560, United States.
  • Department of Biomedical Informatics, Columbia University Medical Center, New York City, NY 10032, United States.
  • Department of Biostatistics, Epidemiology and Informatics (DBEI), The Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, United States.
  • Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90095, United States.
  • VA Informatics and Computing Infrastructure, US Department of Veterans Affairs, Salt Lake City, UT 20420, United States.

ORCID-Profile: Mitchell M Conover, Yong Chen, Marc A Suchard

Förderung

  • NHLBI NIH HHS
  • NLM NIH HHS
  • NIH HHS
  • National Institutes of Health

Zitieren

APA

Conover, M. M., Ryan, P. B., Chen, Y., Suchard, M. A., Hripcsak, G., & Schuemie, M. J. (2025). Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae317

Vancouver

Conover MM, Ryan PB, Chen Y, Suchard MA, Hripcsak G, Schuemie MJ. Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence. Journal of the American Medical Informatics Association. 2025. doi:10.1093/jamia/ocae317

BibTeX
@article{Conover2025,
  author = {Conover, Mitchell M and Ryan, Patrick B and Chen, Yong and Suchard, Marc A and Hripcsak, George and Schuemie, Martijn J},
  title = {Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  doi = {10.1093/jamia/ocae317}
}
Kostenlos starten →