EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Cady Berkel iD, Dillon C Knox, Nikolaos Flemotomos, Victor R Martinez, David C Atkins, Shrikanth S Narayanan, Lizeth Alonso Rodriguez, Carlos G Gallo, Justin D Smith

Implementation Research and Practice · 2023-01 · 4

DOI: 10.1177/26334895231187906 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Implementation Research and Practice, online erschienen am 25.07.2023. Verfasst von Cady Berkel und 8 weiteren Autor:innen. Beteiligte Einrichtung: College of Health Solutions, Arizona State University, Phoenix, AZ, USA. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 77 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Center for Chronic Disease Prevention and Health Promotion.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83–1.02 to 0.62–0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81–27.3 to 0.62–19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Schlagwörter

Stichwörter der Autor:innen

health careimplementationimplementation outcomesintegrated carepreventive interventionprovidertreatment fidelityvalidity

Autor:innen und Institutionen

  • College of Health Solutions, Arizona State University, Phoenix, AZ, USA.
  • Ming Hsieh Department of Electrical Engineering, USC Viterbi School of Engineering, REACH Institute, Arizona State University, Tempe, AZ, USA.
  • Signal Analysis and Interpretation Laboratory, University of Southern California, Los Angeles, CA, USA.
  • Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA.
  • Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago, IL, USA.
  • Department of Population Health Sciences, Spencer Fox Eccles School of Medicine, University of Utah, Salt Lake City, UT, USA.

ORCID-Profile: Cady Berkel

Förderung

  • National Center for Chronic Disease Prevention and Health Promotion

Zitieren

APA

Berkel, C., Knox, D. C., Flemotomos, N., Martinez, V. R., Atkins, D. C., Narayanan, S. S., Rodriguez, L. A., Gallo, C. G., & Smith, J. D. (2023). A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care. Implementation Research and Practice, 4. https://doi.org/10.1177/26334895231187906

Vancouver

Berkel C, Knox DC, Flemotomos N, Martinez VR, Atkins DC, Narayanan SS, et al. A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care. Implementation Research and Practice. 2023;4. doi:10.1177/26334895231187906

BibTeX
@article{Berkel2023,
  author = {Berkel, Cady and Knox, Dillon C and Flemotomos, Nikolaos and Martinez, Victor R and Atkins, David C and Narayanan, Shrikanth S and Rodriguez, Lizeth Alonso and Gallo, Carlos G and Smith, Justin D},
  title = {A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care},
  journal = {Implementation Research and Practice},
  year = {2023},
  volume = {4},
  doi = {10.1177/26334895231187906}
}
Kostenlos starten →