EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research

Jeeva Sam, T Spreuer, M Berger, A Yogeswaran, T Khodr, P Janetzko, W Seeger, J Bienzeisler, R W Majeed

Studies in Health Technology and Informatics · 2026

DOI: 10.3233/shti261007 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2026. Verfasst von Jeeva Sam und 8 weiteren Autor:innen. Beteiligte Einrichtung: Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany. Hauptthemen laut MEDLINE: Benchmarking, Hypertension, Pulmonary, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Statistical analysis of clinical data requires expertise in medical statistics. Large language models (LLMs) are increasingly used for code generation and may support both descriptive and advanced analyses, but their reliability remains uncertain. This study evaluated whether five current LLMs (GPT 5.3, Claude Sonnet 4.6, Gemini 2.5 Flash, Perplexity, and Grok) could reproduce expert-validated statistical analyses from a published clinical workflow across three statistical tasks: descriptive table generation, Kaplan–Meier survival analysis, and Cox proportional hazards modelling. All models received identical datasets and standardised prompts. Their outputs were compared with analyses performed by two experts trained in mathematical statistics and evaluated for grouping correctness, numerical accuracy, missing-value handling, and quality of generated R code. All five models produced correct descriptive statistics once dataset variables were specified explicitly. Two models failed the initial descriptive benchmark because of variable-name ambiguity, but both recovered after prompt clarification. All models reproduced the correct Kaplan–Meier p-value, although figure completeness differed. In the Cox benchmark, four models reproduced all required hazard-ratio terms, while one omitted the interaction results. These findings suggest that LLMs may support clinical research workflows, but their outputs still require careful validation before use.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingHypertension, PulmonaryArtificial IntelligenceLarge Language ModelsHumansProportional Hazards ModelsReproducibility of ResultsKaplan-Meier EstimateData Interpretation, Statistical

Stichwörter der Autor:innen

Artificial IntelligenceHypertensionLarge language modelsProportional Hazards ModelsPulmonaryReproducibility of ResultsSurvival Analysis

Autor:innen und Institutionen

  • Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany.
  • Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany.
  • Institute of Medical Informatics, RWTH Aachen University Medical Faculty, Aachen, Germany.

Zitieren

APA

Sam, J., Spreuer, T., Berger, M., Yogeswaran, A., Khodr, T., Janetzko, P., Seeger, W., Bienzeisler, J., & Majeed, R. W. (2026). Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti261007

Vancouver

Sam J, Spreuer T, Berger M, Yogeswaran A, Khodr T, Janetzko P, et al. Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research. Studies in Health Technology and Informatics. 2026. doi:10.3233/shti261007

BibTeX
@article{Sam2026,
  author = {Sam, Jeeva and Spreuer, T and Berger, M and Yogeswaran, A and Khodr, T and Janetzko, P and Seeger, W and Bienzeisler, J and Majeed, R W},
  title = {Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research},
  journal = {Studies in Health Technology and Informatics},
  year = {2026},
  doi = {10.3233/shti261007}
}
Kostenlos starten →