EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating large language models for automated REDCap support ticket triage and response

Andrew Carroll, Dawn Brown, Domenick Silvio, James Maszatics, David A Hanauer

JAMIA Open · 05.05.2026 · 9 (3)

DOI: 10.1093/jamiaopen/ooag073 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 05.06.2026. Verfasst von Andrew Carroll und 4 weiteren Autor:innen. Beteiligte Einrichtung: Michigan Institute for Clinical and Health Research, University of Michigan Medical School, Ann Arbor, MI, 48109, United States. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NCATS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To evaluate the utility of large language models (LLMs) without and with retrieval-augmented generation (RAG) resources for automating Research Electronic Data Capture (REDCap) support at a large academic medical center, focusing on ticket complexity, categorization, and response drafting. Materials and Methods We randomly selected 90 REDCap support tickets from 6316 received in 2023, stratified by estimated complexity (low, moderate, high) using ticket metadata. Four LLM configurations (GPT-4o Default; GPT-4o + Vanderbilt REDCap documentation; GPT-4o + University of Michigan [UM] REDCap documentation; GPT-4o + combined documentation) were evaluated on 3 tasks per ticket: complexity assessment, category assignment, and email response drafting. Human REDCap experts provided gold-standard ratings and assessed email drafts.

Results

Compared to the gold standard, the GPT-4o Default environment most closely matched human complexity ratings, while specialized RAG environments tended to overestimate complexity. For ticket support categorization, all LLM configurations achieved high agreement with human judgments (Cohen’s kappa ≈0.85), correctly categorizing 81%-89% of tickets. In generating ticket responses, the GPT-4o + REDCap + UM environment provided the most correct answers, particularly for low and moderate complexity tickets, where about two-thirds of responses were substantially or fully correct.

Discussion

Large language models have potential to automate ticket triage and response drafting for routine REDCap support, with context augmentation improving response accuracy for lower complexity tickets.

Conclusion

Large language model-based support tools hold promise for reducing human workload for REDCap support tickets. Continued human oversight, iterative prompt engineering, and expanded local documentation are needed for reliable, effective deployment.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencehealth information systemslarge language modelsnatural language processingoffice automation

Autor:innen und Institutionen

  • Michigan Institute for Clinical and Health Research, University of Michigan Medical School, Ann Arbor, MI, 48109, United States.
  • Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, 48109, United States.

Förderung

  • NCATS NIH HHS

Zitieren

APA

Carroll, A., Brown, D., Silvio, D., Maszatics, J., & Hanauer, D. A. (2026). Evaluating large language models for automated REDCap support ticket triage and response. JAMIA Open, 9(3). https://doi.org/10.1093/jamiaopen/ooag073

Vancouver

Carroll A, Brown D, Silvio D, Maszatics J, Hanauer DA. Evaluating large language models for automated REDCap support ticket triage and response. JAMIA Open. 2026;9(3). doi:10.1093/jamiaopen/ooag073

BibTeX
@article{Carroll2026,
  author = {Carroll, Andrew and Brown, Dawn and Silvio, Domenick and Maszatics, James and Hanauer, David A},
  title = {Evaluating large language models for automated REDCap support ticket triage and response},
  journal = {JAMIA Open},
  year = {2026},
  volume = {9},
  number = {3},
  doi = {10.1093/jamiaopen/ooag073}
}
Kostenlos starten →