EUVIMEDEUVIMED entdecken
Open Access

Automating Motivational Interviewing Coding in Adolescent Substance Use Prevention: Human-AI Agreement Study

Francisco Cardozo, Eric C Brown, Juliana Mejía-Trujillo, Raymond Balise, Catalina Cañizares, Augusto Pérez-Gómez, Sara M St George, Vilma Gabbay

JMIR AI · 25.08.2026 · 5 · e95964-e95964

DOI: 10.2196/95964 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 25.08.2026. Verfasst von Francisco Cardozo und 7 weiteren Autor:innen. Beteiligte Einrichtung: Miller School of Medicine, University of Miami, 1252 Memorial Dr, Coral Gables, Miami, FL, 33146, United States, 1 305-284-2211. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS, NIDA NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Motivational interviewing (MI) is widely used in preventive interventions, yet coding MI techniques and monitoring intervention adherence remain resource-intensive due to the reliance on manual transcription and expert review. Large language models (LLMs) offer a promising approach to automate these tasks, but their agreement with human coders in the context of prevention interventions has not been established.

Objective

This study evaluated the agreement between an AI-based coder (OpenAI’s GPT 4.1) and trained human coders on two tasks: (1) identification of MI techniques (eg, open questions, affirmations, giving information) at the facilitator-message level and (2) completing a 21-item checklist of implementation adherence for a brief MI-based preventive intervention for adolescent substance use.

Methods

Two certified MI facilitators independently coded 72 facilitator messages from 2 standardized Spanish-language Brief Intervention Based on Motivational Interviewing program (Intervención Breve Basada en Entrevista Motivacional [IBEM]) practice sessions with chatbot-simulated adolescent responses. The facilitators classified MI techniques using the OARS (open questions, affirmations, reflections, and summaries) framework and completed a 21-item implementation-adherence checklist. An AI-based coder (OpenAI’s GPT-4.1, accessed through the API) classified the same facilitator messages and checklist items using a structured prompt derived from the MI coding manual. MI techniques were compared at the facilitator-message level and implementation adherence at the session level. Intercoder agreement in use of MI techniques and implementation adherence was assessed using Cohen κ, Fleiss κ, and Cochran Q tests.

Results

For use of MI techniques, the AI coder demonstrated moderate-to-substantial agreement with human coders across most techniques, including open questions (κ=0.66-0.69), affirmations (κ=0.66-0.77), and giving information (κ=0.91). No statistically significant differences in percentages of MI technique use were observed among the 3 coders, although agreement was the weakest for higher-inference categories such as complex reflections (κ=0.00). For MI implementation adherence, overall agreement was moderate (Fleiss κ=0.487), and pairwise agreement between the AI coder and 1 human coder was substantial (κ=0.67), exceeding the agreement observed between the 2 human coders (κ=0.53).

Conclusions

These findings provide support for the feasibility of using LLMs to recognize MI techniques and assess implementation adherence. The results support a human-AI collaborative model in which the AI coder “precodes” facilitator messages and flags sessions for expert review, while human coders retain responsibility for higher-inference judgments and shift their effort from routine coding toward contextual review and coaching feedback. Because the analyses are based on only 2 sessions, these results should be interpreted as early-stage, proof-of-concept evidence rather than a basis for large-scale deployment. Future research should compare different LLMs and evaluate whether AI-assisted coding improves the scalability of routine implementation monitoring.

Schlagwörter

Stichwörter der Autor:innen

adherenceartificial intelligenceimplementation fidelityintercoder agreementlarge language modelsmotivational interviewingpreventive interventions

Autor:innen und Institutionen

  • Miller School of Medicine, University of Miami, 1252 Memorial Dr, Coral Gables, Miami, FL, 33146, United States, 1 305-284-2211.
  • Corporación Nuevos Rumbos, Bogotá, Colombia.
  • Applied Psychology, New York University, New York, NY, United States.
  • Nathan Kline Institute for Psychiatric Research, Orangeburg, NY, United States.

Förderung

  • NIMH NIH HHS
  • NIDA NIH HHS

Zitieren

APA

Cardozo, F., Brown, E. C., Mejía-Trujillo, J., Balise, R., Cañizares, C., Pérez-Gómez, A., St George, S. M., & Gabbay, V. (2026). Automating Motivational Interviewing Coding in Adolescent Substance Use Prevention: Human-AI Agreement Study. JMIR AI, 5, e95964-e95964. https://doi.org/10.2196/95964

Vancouver

Cardozo F, Brown EC, Mejía-Trujillo J, Balise R, Cañizares C, Pérez-Gómez A, et al. Automating Motivational Interviewing Coding in Adolescent Substance Use Prevention: Human-AI Agreement Study. JMIR AI. 2026;5:e95964-e95964. doi:10.2196/95964

BibTeX
@article{Cardozo2026,
  author = {Cardozo, Francisco and Brown, Eric C and Mejía-Trujillo, Juliana and Balise, Raymond and Cañizares, Catalina and Pérez-Gómez, Augusto and St George, Sara M and Gabbay, Vilma},
  title = {Automating Motivational Interviewing Coding in Adolescent Substance Use Prevention: Human-AI Agreement Study},
  journal = {JMIR AI},
  year = {2026},
  volume = {5},
  pages = {e95964--e95964},
  doi = {10.2196/95964}
}
Kostenlos starten →