Assessing the response consistency in Chat Generative Pretrained Transformer: The case of “What is Ayurveda?”
International Journal of Health & Allied Sciences · 2025-01 · 14 (1) · 77-82
DOI: 10.4103/ijhas.ijhas_64_25 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Health & Allied Sciences, online erschienen am 01.10.2025. Verfasst von Janmejaya Samal. Die Arbeit nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).
Abstract (Originalsprache)
INTRODUCTION: Chat Generative Pretrained Transformer (ChatGPT) is a language-based chatbot launched in November 2022. This study was carried out to understand the consistency of the responses of ChatGPT to a single question when asked 10 different times, consecutively, within a particular time frame. MATERIALS AND METHODS: The study followed two steps: in the first step, only one question, “What is Ayurveda ?” was asked, followed by a conversation for bibliographic citations. The responses obtained were analyzed statistically, and the key concepts from the responses were free listed. Statistical analysis was conducted using Jamovi (version 2.3.28). RESULTS: The median word count was 334 (standard deviation [SD] = 32.87, interquartile range [IQR] =313–354), sentences were 18 (SD = 2.36, IQR = 17.0–20.3), headings were 1.50 (SD = 0.82, IQR = 1–2), and the subheadings were 8 (SD = 2.08, IQR = 7–9). The linear regression between the number of words and other particulars showed a statistically significant association with the number of sentences ( P = 0.003, 95% confidence interval). The free listing of the key concepts was done under three headings: core principles, treatment practices, and modern relevance. Four concepts were found to have the highest frequencies in the ChatGPT responses: Doshas ( n = 10, 100%), Panchakarma ( n = 7, 70%), Pancha Mahabhutas ( n = 7, 70%), and herbal medicine ( n = 6, 60%). In the conversation, ChatGPT provided 21 different bibliographic citations with three rounds of questions; however, none of the citations were found to exist. CONCLUSIONS: ChatGPT can be useful in many ways in education and research. However, it should be used with a pinch of salt as ChatGPT itself says, “ChatGPT can make mistakes. Check important info.”
Zitieren
APA
Samal, J. (2025). Assessing the response consistency in Chat Generative Pretrained Transformer: The case of “What is Ayurveda?”. International Journal of Health & Allied Sciences, 14(1), 77-82. https://doi.org/10.4103/ijhas.ijhas_64_25
Vancouver
Samal J. Assessing the response consistency in Chat Generative Pretrained Transformer: The case of “What is Ayurveda?”. International Journal of Health & Allied Sciences. 2025;14(1):77-82. doi:10.4103/ijhas.ijhas_64_25
BibTeX
@article{Samal2025,
author = {Samal, Janmejaya},
title = {Assessing the response consistency in Chat Generative Pretrained Transformer: The case of “What is Ayurveda?”},
journal = {International Journal of Health & Allied Sciences},
year = {2025},
volume = {14},
number = {1},
pages = {77--82},
doi = {10.4103/ijhas.ijhas_64_25}
}
EUVIMED