Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study
Journal of Clinical Medicine · 20.12.2024 · 13 (24) · 7810
DOI: 10.3390/jcm13247810 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 20.12.2024. Verfasst von Valentina Ponzo und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Science, University of Turin, 10126 Torino, Italy. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
The use of artificial intelligence (AI) chatbots for obtaining healthcare advice is greatly increased in the general population. This study assessed the performance of general-purpose AI chatbots in giving nutritional advice for patients with obesity with or without multiple comorbidities.
Methods
The case of a 35-year-old male with obesity without comorbidities (Case 1), and the case of a 65-year-old female with obesity, type 2 diabetes mellitus, sarcopenia, and chronic kidney disease (Case 2) were submitted to 10 different AI chatbots on three consecutive days. Accuracy (the ability to provide advice aligned with guidelines), completeness, and reproducibility (replicability of the information over the three days) of the chatbots’ responses were evaluated by three registered dietitians. Nutritional consistency was evaluated by comparing the nutrient content provided by the chatbots with values calculated by dietitians.
Results
Case 1: ChatGPT 3.5 demonstrated the highest accuracy rate (67.2%) and Copilot the lowest (21.1%). ChatGPT 3.5 and ChatGPT 4.0 achieved the highest completeness (both 87.3%), whereas Gemini and Copilot recorded the lowest scores (55.6%, 42.9%, respectively). Reproducibility was highest for Chatsonic (86.1%) and lowest for ChatGPT 4.0 (50%) and ChatGPT 3.5 (52.8%). Case 2: Overall accuracy was low, with no chatbot achieving 50% accuracy. Completeness was highest for ChatGPT 4.0 and Claude (both 77.8%), and lowest for Copilot (23.3%). ChatGPT 4.0 and Pi Ai showed the lowest reproducibility. Major inconsistencies regarded the amount of protein recommended by most chatbots, which suggested simultaneously to both reduce and increase protein intake.
Conclusions
General-purpose AI chatbots exhibited limited accuracy, reproducibility, and consistency in giving dietary advice in complex clinical scenarios and cannot replace the work of an expert dietitian.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medical Science, University of Turin, 10126 Torino, Italy.
- Department of Psychology, University of Turin, 10124 Torino, Italy.
- Dietetic and Clinical Nutrition Unit, Città della Salute e della Scienza Hospital, 10126 Torino, Italy.
- Department of Food Science and Technology, University of Gastronomic Sciences, 12042 Pollenzo, Italy.
- Department of Translatioal Medicine, University of Ferrara, 44121 Ferrara, Italy.
ORCID-Profile: Rosalba Rosato, Andrea Devecchi, Fabio Dario Merlo, Domenico Sergi, Simona Bo
Zitieren
APA
Ponzo, V., Rosato, R., Scigliano, M. C., Onida, M., Cossai, S., De Vecchi, M., Devecchi, A., Goitre, I., Favaro, E., Merlo, F. D., Sergi, D., & Bo, S. (2024). Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study. Journal of Clinical Medicine, 13(24), 7810. https://doi.org/10.3390/jcm13247810
Vancouver
Ponzo V, Rosato R, Scigliano MC, Onida M, Cossai S, De Vecchi M, et al. Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study. Journal of Clinical Medicine. 2024;13(24):7810. doi:10.3390/jcm13247810
BibTeX
@article{Ponzo2024,
author = {Ponzo, Valentina and Rosato, Rosalba and Scigliano, Maria Carmine and Onida, Martina and Cossai, Simona and De Vecchi, Morena and Devecchi, Andrea and Goitre, Ilaria and Favaro, Enrica and Merlo, Fabio Dario and Sergi, Domenico and Bo, Simona},
title = {Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study},
journal = {Journal of Clinical Medicine},
year = {2024},
volume = {13},
number = {24},
pages = {7810},
doi = {10.3390/jcm13247810}
}
EUVIMED