EUVIMEDOpdag EUVIMED
ValidierungsstudieOpen access

An Extraction Tool for Venous Thromboembolism Symptom Identification in Primary Care Notes to Facilitate Electronic Clinical Quality Measure Reporting: Algorithm Development and Validation Study

John Novoa-Laurentiev iD, Mica Bowen iD, Avery Pullman iD, Wenyu Song iD, Ania Syrowatka iD, Jin Chen iD, Michael Sainlaire, Frank Chang iD, Krissy Gray iD, Purushottam Panta iD, Luwei Liu iD, Khalid Nawab iD and 4 more

JMIR Medical Informatics · 2025

DOI: 10.2196/63720 ↗

At a glance

Validierungsstudie in JMIR Medical Informatics, 2025. Written by John Novoa-Laurentiev and 15 co-authors. Institution: Department of Medicine, Brigham & Women's Hospital, 75 Francis Street, Boston, MA, 02115, United States, 1 8572824088. Major MEDLINE topics: Venous Thromboembolism, Primary Health Care, Natural Language Processing, Electronic Health Records. Cited 5 times so far (Crossref); 69 references. The full text is freely available in PubMed Central. Funded by: NCATS NIH HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Diagnosis of venous thromboembolism (VTE) is often delayed, and facilitating earlier diagnosis may improve associated morbidity and mortality. Clinical notes contain information not found elsewhere in the medical record that could facilitate timely VTE diagnosis and accurate quality measurement. However, extracting relevant information from unstructured clinical notes is complex. Today, there are relatively few electronic clinical quality measures (eCQMs) in our national payment program and none that use natural language processing (NLP) techniques for data extraction. NLP holds great promise for making quality measurement more accurate and more efficient. Given the potential of NLP-based applications to facilitate more accurate VTE detection, primary care is one clinical setting in urgent need of this type of tool.

Objective

This study aimed to develop a tool that extracts VTE symptoms from clinical notes for use within an eCQM to quantify the rate of delayed diagnosis of VTE in primary care settings.

Methods

We iteratively developed an NLP-based data extraction tool, venous thromboembolism symptom extractor (VTExt), on an internal dataset using a rule-based approach to extract VTE symptoms from primary care clinical note text. The VTE symptoms lexicon was derived and optimized with physician guidance and externally validated using datasets from 2 independent health care organizations. We performed 26 rounds of performance evaluation of notes sampled from the case cohort (17,585 patient progress note sentences from 279 patient notes), and 5 rounds of evaluation of the control cohort (2838 patient progress note sentences from 50 patient notes). VTExt’s performance was evaluated using evaluation metrics, including area under the curve, positive predictive value, negative predictive value, sensitivity, and specificity.

Results

VTExt achieved near-perfect performance in extracting VTE symptoms from primary care notes sampled from records of patients diagnosed with or without VTE. In external validation, VTExt achieved promising performance in 2 additional geographically distant organizations using different electronic health record systems. When compared against a deep learning model and 4 machine learning models, VTExt exhibited similar or even improved performance across all metrics.

Conclusions

This study demonstrates a data-driven NLP-based approach to clinical note information extraction that can be generalized to different electronic health record systems across different institutions. Due to the robust performance of this tool, VTExt is the first NLP application to be used in a nationally endorsed eCQM.

Subjects

MeSH terms (MEDLINE)

Venous ThromboembolismPrimary Health CareNatural Language ProcessingElectronic Health RecordsAlgorithmsQuality Indicators, Health CareHumansFemaleMaleMiddle Aged

Author keywords

AINLPVTEalgorithm developmentalgorithm validationartificial intelligenceclinical noteclinical qualitydiagnosiselectronic clinical quality measureextractionextraction toolnatural language processingprimary carethromboembolism

Authors and institutions

  • Department of Medicine, Brigham & Women's Hospital, 75 Francis Street, Boston, MA, 02115, United States, 1 8572824088.
  • Department of Medicine, Harvard Medical School, Boston, MA, United States.
  • Department of Medicine, University of Alabama at Birmingham, Birmingham, AL, United States.
  • Department of Biomedical Informatics, University of Kentucky, Lexington, KY, United States.
  • Department of Information Services, Penn State Health, Hershey, PA, United States.
  • Department of Medicine, Penn State College of Medicine, Hershey, PA, United States.

ORCID profiles: John Novoa-Laurentiev, Mica Bowen, Avery Pullman, Wenyu Song, Ania Syrowatka, Jin Chen, Frank Chang, Krissy Gray, Purushottam Panta, Luwei Liu, Khalid Nawab, Shadi Hijjawi, Richard Schreiber, Li Zhou, Patricia C Dykes

Funding

  • NCATS NIH HHS

Cite

APA

Novoa-Laurentiev, J., Bowen, M., Pullman, A., Song, W., Syrowatka, A., Chen, J., Sainlaire, M., Chang, F., Gray, K., Panta, P., Liu, L., Nawab, K., Hijjawi, S., Schreiber, R., Zhou, L., & Dykes, P. C. (2025). An Extraction Tool for Venous Thromboembolism Symptom Identification in Primary Care Notes to Facilitate Electronic Clinical Quality Measure Reporting: Algorithm Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/63720

Vancouver

Novoa-Laurentiev J, Bowen M, Pullman A, Song W, Syrowatka A, Chen J, et al. An Extraction Tool for Venous Thromboembolism Symptom Identification in Primary Care Notes to Facilitate Electronic Clinical Quality Measure Reporting: Algorithm Development and Validation Study. JMIR Medical Informatics. 2025. doi:10.2196/63720

BibTeX
@article{NovoaLaurentiev2025,
  author = {Novoa-Laurentiev, John and Bowen, Mica and Pullman, Avery and Song, Wenyu and Syrowatka, Ania and Chen, Jin and Sainlaire, Michael and Chang, Frank and Gray, Krissy and Panta, Purushottam and Liu, Luwei and Nawab, Khalid and Hijjawi, Shadi and Schreiber, Richard and Zhou, Li and Dykes, Patricia C},
  title = {An Extraction Tool for Venous Thromboembolism Symptom Identification in Primary Care Notes to Facilitate Electronic Clinical Quality Measure Reporting: Algorithm Development and Validation Study},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/63720}
}
Start for free →