Evaluating large language models for automated REDCap support ticket triage and response
JAMIA Open · 05.05.2026 · 9 (3)
DOI: 10.1093/jamiaopen/ooag073 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 05.06.2026. Verfasst von Andrew Carroll und 4 weiteren Autor:innen. Beteiligte Einrichtung: Michigan Institute for Clinical and Health Research, University of Michigan Medical School, Ann Arbor, MI, 48109, United States. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NCATS NIH HHS.
Abstract (Originalsprache)
Objectives
To evaluate the utility of large language models (LLMs) without and with retrieval-augmented generation (RAG) resources for automating Research Electronic Data Capture (REDCap) support at a large academic medical center, focusing on ticket complexity, categorization, and response drafting. Materials and Methods We randomly selected 90 REDCap support tickets from 6316 received in 2023, stratified by estimated complexity (low, moderate, high) using ticket metadata. Four LLM configurations (GPT-4o Default; GPT-4o + Vanderbilt REDCap documentation; GPT-4o + University of Michigan [UM] REDCap documentation; GPT-4o + combined documentation) were evaluated on 3 tasks per ticket: complexity assessment, category assignment, and email response drafting. Human REDCap experts provided gold-standard ratings and assessed email drafts.
Results
Compared to the gold standard, the GPT-4o Default environment most closely matched human complexity ratings, while specialized RAG environments tended to overestimate complexity. For ticket support categorization, all LLM configurations achieved high agreement with human judgments (Cohen’s kappa ≈0.85), correctly categorizing 81%-89% of tickets. In generating ticket responses, the GPT-4o + REDCap + UM environment provided the most correct answers, particularly for low and moderate complexity tickets, where about two-thirds of responses were substantially or fully correct.
Discussion
Large language models have potential to automate ticket triage and response drafting for routine REDCap support, with context augmentation improving response accuracy for lower complexity tickets.
Conclusion
Large language model-based support tools hold promise for reducing human workload for REDCap support tickets. Continued human oversight, iterative prompt engineering, and expanded local documentation are needed for reliable, effective deployment.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Michigan Institute for Clinical and Health Research, University of Michigan Medical School, Ann Arbor, MI, 48109, United States.
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, 48109, United States.
Förderung
- NCATS NIH HHS
Zitieren
APA
Carroll, A., Brown, D., Silvio, D., Maszatics, J., & Hanauer, D. A. (2026). Evaluating large language models for automated REDCap support ticket triage and response. JAMIA Open, 9(3). https://doi.org/10.1093/jamiaopen/ooag073
Vancouver
Carroll A, Brown D, Silvio D, Maszatics J, Hanauer DA. Evaluating large language models for automated REDCap support ticket triage and response. JAMIA Open. 2026;9(3). doi:10.1093/jamiaopen/ooag073
BibTeX
@article{Carroll2026,
author = {Carroll, Andrew and Brown, Dawn and Silvio, Domenick and Maszatics, James and Hanauer, David A},
title = {Evaluating large language models for automated REDCap support ticket triage and response},
journal = {JAMIA Open},
year = {2026},
volume = {9},
number = {3},
doi = {10.1093/jamiaopen/ooag073}
}
EUVIMED