Evaluating large language models for selection of statistical test for research: A pilot study
Perspectives in Clinical Research · 2024-10 · 15 (4) · 178-182
DOI: 10.4103/picr.picr_275_23 ↗Kurz gefasst
Fachartikel in der Zeitschrift Perspectives in Clinical Research, online erschienen am 08.04.2024. Verfasst von Himel Mondal und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 12 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
In contemporary research, selecting the appropriate statistical test is a critical and often challenging step. The emergence of large language models (LLMs) has offered a promising avenue for automating this process, potentially enhancing the efficiency and accuracy of statistical test selection.
Aim
This study aimed to assess the capability of freely available LLMs – OpenAI’s ChatGPT3.5, Google Bard, Microsoft Bing Chat, and Perplexity in recommending suitable statistical tests for research, comparing their recommendations with those made by human experts. Materials and Methods: A total of 27 case vignettes were prepared for common research models with a question asking suitable statistical tests. The cases were formulated from previously published literature and reviewed by a human expert for their accuracy of information. The LLMs were asked the question with the case vignettes and the process was repeated with paraphrased cases. The concordance (if exactly matching the answer key) and acceptance (when not exactly matching with answer key, but can be considered suitable) were evaluated between LLM’s recommendations and those of human experts.
Results
Among the 27 case vignettes, ChatGPT3.5-suggested statistical test had 85.19% concordance and 100% acceptance; Bard experiment had 77.78% concordance and 96.3% acceptance; Microsoft Bing Chat had 96.3% concordance and 100% acceptance; and Perplexity had 85.19% concordance and 100% acceptance. The intra-class correction coefficient of average measure among the responses of LLMs was 0.728 (95% confidence interval [CI]: 0.51–0.86), P < 0.0001. The test–retest reliability of ChatGPT was r = 0.71 (95% CI: 0.44–0.86), P < 0.0001, Bard was r = −0.22 (95% CI: −0.56–0.18), P = 0.26, Bing was r = −0.06 (95% CI: −0.44–0.33), P = 0.73, and Perplexity was r = 0.52 (95% CI: 0.16–0.75), P = 0.0059.
Conclusion
The LLMs, namely, ChatGPT, Google Bard, Microsoft Bing, and Perplexity all showed >75% concordance in suggesting statistical tests for research case vignettes with all having acceptance of >95%. The LLMs had a moderate level of agreement among them. While not a complete replacement for human expertise, these models can serve as effective decision support systems, especially in scenarios where rapid test selection is essential.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India.
- Department of Physiology, Raiganj Government Medical College and Hospital, West Bengal, India.
- Department of Business Data Processing and Management, Satyawati College (Eve.), University of Delhi, New Delhi, India.
Zitieren
APA
Mondal, H., Mondal, S., & Mittal, P. (2024). Evaluating large language models for selection of statistical test for research: A pilot study. Perspectives in Clinical Research, 15(4), 178-182. https://doi.org/10.4103/picr.picr_275_23
Vancouver
Mondal H, Mondal S, Mittal P. Evaluating large language models for selection of statistical test for research: A pilot study. Perspectives in Clinical Research. 2024;15(4):178-182. doi:10.4103/picr.picr_275_23
BibTeX
@article{Mondal2024,
author = {Mondal, Himel and Mondal, Shaikat and Mittal, Prabhat},
title = {Evaluating large language models for selection of statistical test for research: A pilot study},
journal = {Perspectives in Clinical Research},
year = {2024},
volume = {15},
number = {4},
pages = {178--182},
doi = {10.4103/picr.picr_275_23}
}
EUVIMED