EUVIMEDEUVIMED entdecken
Open Access

Evaluating large language models for selection of statistical test for research: A pilot study

Himel Mondal, Shaikat Mondal, Prabhat Mittal

Perspectives in Clinical Research · 2024-10 · 15 (4) · 178-182

DOI: 10.4103/picr.picr_275_23 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Perspectives in Clinical Research, online erschienen am 08.04.2024. Verfasst von Himel Mondal und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 12 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

In contemporary research, selecting the appropriate statistical test is a critical and often challenging step. The emergence of large language models (LLMs) has offered a promising avenue for automating this process, potentially enhancing the efficiency and accuracy of statistical test selection.

Aim

This study aimed to assess the capability of freely available LLMs – OpenAI’s ChatGPT3.5, Google Bard, Microsoft Bing Chat, and Perplexity in recommending suitable statistical tests for research, comparing their recommendations with those made by human experts. Materials and Methods: A total of 27 case vignettes were prepared for common research models with a question asking suitable statistical tests. The cases were formulated from previously published literature and reviewed by a human expert for their accuracy of information. The LLMs were asked the question with the case vignettes and the process was repeated with paraphrased cases. The concordance (if exactly matching the answer key) and acceptance (when not exactly matching with answer key, but can be considered suitable) were evaluated between LLM’s recommendations and those of human experts.

Results

Among the 27 case vignettes, ChatGPT3.5-suggested statistical test had 85.19% concordance and 100% acceptance; Bard experiment had 77.78% concordance and 96.3% acceptance; Microsoft Bing Chat had 96.3% concordance and 100% acceptance; and Perplexity had 85.19% concordance and 100% acceptance. The intra-class correction coefficient of average measure among the responses of LLMs was 0.728 (95% confidence interval [CI]: 0.51–0.86), P < 0.0001. The test–retest reliability of ChatGPT was r = 0.71 (95% CI: 0.44–0.86), P < 0.0001, Bard was r = −0.22 (95% CI: −0.56–0.18), P = 0.26, Bing was r = −0.06 (95% CI: −0.44–0.33), P = 0.73, and Perplexity was r = 0.52 (95% CI: 0.16–0.75), P = 0.0059.

Conclusion

The LLMs, namely, ChatGPT, Google Bard, Microsoft Bing, and Perplexity all showed >75% concordance in suggesting statistical tests for research case vignettes with all having acceptance of >95%. The LLMs had a moderate level of agreement among them. While not a complete replacement for human expertise, these models can serve as effective decision support systems, especially in scenarios where rapid test selection is essential.

Schlagwörter

Stichwörter der Autor:innen

Accuracyartificial intelligencedata analysislarge language modelsresearch methodologystatistical test selection

Autor:innen und Institutionen

  • Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India.
  • Department of Physiology, Raiganj Government Medical College and Hospital, West Bengal, India.
  • Department of Business Data Processing and Management, Satyawati College (Eve.), University of Delhi, New Delhi, India.

Zitieren

APA

Mondal, H., Mondal, S., & Mittal, P. (2024). Evaluating large language models for selection of statistical test for research: A pilot study. Perspectives in Clinical Research, 15(4), 178-182. https://doi.org/10.4103/picr.picr_275_23

Vancouver

Mondal H, Mondal S, Mittal P. Evaluating large language models for selection of statistical test for research: A pilot study. Perspectives in Clinical Research. 2024;15(4):178-182. doi:10.4103/picr.picr_275_23

BibTeX
@article{Mondal2024,
  author = {Mondal, Himel and Mondal, Shaikat and Mittal, Prabhat},
  title = {Evaluating large language models for selection of statistical test for research: A pilot study},
  journal = {Perspectives in Clinical Research},
  year = {2024},
  volume = {15},
  number = {4},
  pages = {178--182},
  doi = {10.4103/picr.picr_275_23}
}
Kostenlos starten →