Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research
Studies in Health Technology and Informatics · 2026
DOI: 10.3233/shti261007 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2026. Verfasst von Jeeva Sam und 8 weiteren Autor:innen. Beteiligte Einrichtung: Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany. Hauptthemen laut MEDLINE: Benchmarking, Hypertension, Pulmonary, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Statistical analysis of clinical data requires expertise in medical statistics. Large language models (LLMs) are increasingly used for code generation and may support both descriptive and advanced analyses, but their reliability remains uncertain. This study evaluated whether five current LLMs (GPT 5.3, Claude Sonnet 4.6, Gemini 2.5 Flash, Perplexity, and Grok) could reproduce expert-validated statistical analyses from a published clinical workflow across three statistical tasks: descriptive table generation, Kaplan–Meier survival analysis, and Cox proportional hazards modelling. All models received identical datasets and standardised prompts. Their outputs were compared with analyses performed by two experts trained in mathematical statistics and evaluated for grouping correctness, numerical accuracy, missing-value handling, and quality of generated R code. All five models produced correct descriptive statistics once dataset variables were specified explicitly. Two models failed the initial descriptive benchmark because of variable-name ambiguity, but both recovered after prompt clarification. All models reproduced the correct Kaplan–Meier p-value, although figure completeness differed. In the Cox benchmark, four models reproduced all required hazard-ratio terms, while one omitted the interaction results. These findings suggest that LLMs may support clinical research workflows, but their outputs still require careful validation before use.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany.
- Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany.
- Institute of Medical Informatics, RWTH Aachen University Medical Faculty, Aachen, Germany.
Zitieren
APA
Sam, J., Spreuer, T., Berger, M., Yogeswaran, A., Khodr, T., Janetzko, P., Seeger, W., Bienzeisler, J., & Majeed, R. W. (2026). Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti261007
Vancouver
Sam J, Spreuer T, Berger M, Yogeswaran A, Khodr T, Janetzko P, et al. Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research. Studies in Health Technology and Informatics. 2026. doi:10.3233/shti261007
BibTeX
@article{Sam2026,
author = {Sam, Jeeva and Spreuer, T and Berger, M and Yogeswaran, A and Khodr, T and Janetzko, P and Seeger, W and Bienzeisler, J and Majeed, R W},
title = {Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research},
journal = {Studies in Health Technology and Informatics},
year = {2026},
doi = {10.3233/shti261007}
}
EUVIMED