EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study

Valentina Ponzo, Rosalba Rosato iD, Maria Carmine Scigliano, Martina Onida, Simona Cossai, Morena De Vecchi, Andrea Devecchi iD, Ilaria Goitre, Enrica Favaro, Fabio Dario Merlo iD, Domenico Sergi iD, Simona Bo iD

Journal of Clinical Medicine · 20.12.2024 · 13 (24) · 7810

DOI: 10.3390/jcm13247810 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 20.12.2024. Verfasst von Valentina Ponzo und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Science, University of Turin, 10126 Torino, Italy. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The use of artificial intelligence (AI) chatbots for obtaining healthcare advice is greatly increased in the general population. This study assessed the performance of general-purpose AI chatbots in giving nutritional advice for patients with obesity with or without multiple comorbidities.

Methods

The case of a 35-year-old male with obesity without comorbidities (Case 1), and the case of a 65-year-old female with obesity, type 2 diabetes mellitus, sarcopenia, and chronic kidney disease (Case 2) were submitted to 10 different AI chatbots on three consecutive days. Accuracy (the ability to provide advice aligned with guidelines), completeness, and reproducibility (replicability of the information over the three days) of the chatbots’ responses were evaluated by three registered dietitians. Nutritional consistency was evaluated by comparing the nutrient content provided by the chatbots with values calculated by dietitians.

Results

Case 1: ChatGPT 3.5 demonstrated the highest accuracy rate (67.2%) and Copilot the lowest (21.1%). ChatGPT 3.5 and ChatGPT 4.0 achieved the highest completeness (both 87.3%), whereas Gemini and Copilot recorded the lowest scores (55.6%, 42.9%, respectively). Reproducibility was highest for Chatsonic (86.1%) and lowest for ChatGPT 4.0 (50%) and ChatGPT 3.5 (52.8%). Case 2: Overall accuracy was low, with no chatbot achieving 50% accuracy. Completeness was highest for ChatGPT 4.0 and Claude (both 77.8%), and lowest for Copilot (23.3%). ChatGPT 4.0 and Pi Ai showed the lowest reproducibility. Major inconsistencies regarded the amount of protein recommended by most chatbots, which suggested simultaneously to both reduce and increase protein intake.

Conclusions

General-purpose AI chatbots exhibited limited accuracy, reproducibility, and consistency in giving dietary advice in complex clinical scenarios and cannot replace the work of an expert dietitian.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencechatbotsdietary advicedietary plansobesity

Autor:innen und Institutionen

  • Department of Medical Science, University of Turin, 10126 Torino, Italy.
  • Department of Psychology, University of Turin, 10124 Torino, Italy.
  • Dietetic and Clinical Nutrition Unit, Città della Salute e della Scienza Hospital, 10126 Torino, Italy.
  • Department of Food Science and Technology, University of Gastronomic Sciences, 12042 Pollenzo, Italy.
  • Department of Translatioal Medicine, University of Ferrara, 44121 Ferrara, Italy.

ORCID-Profile: Rosalba Rosato, Andrea Devecchi, Fabio Dario Merlo, Domenico Sergi, Simona Bo

Zitieren

APA

Ponzo, V., Rosato, R., Scigliano, M. C., Onida, M., Cossai, S., De Vecchi, M., Devecchi, A., Goitre, I., Favaro, E., Merlo, F. D., Sergi, D., & Bo, S. (2024). Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study. Journal of Clinical Medicine, 13(24), 7810. https://doi.org/10.3390/jcm13247810

Vancouver

Ponzo V, Rosato R, Scigliano MC, Onida M, Cossai S, De Vecchi M, et al. Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study. Journal of Clinical Medicine. 2024;13(24):7810. doi:10.3390/jcm13247810

BibTeX
@article{Ponzo2024,
  author = {Ponzo, Valentina and Rosato, Rosalba and Scigliano, Maria Carmine and Onida, Martina and Cossai, Simona and De Vecchi, Morena and Devecchi, Andrea and Goitre, Ilaria and Favaro, Enrica and Merlo, Fabio Dario and Sergi, Domenico and Bo, Simona},
  title = {Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study},
  journal = {Journal of Clinical Medicine},
  year = {2024},
  volume = {13},
  number = {24},
  pages = {7810},
  doi = {10.3390/jcm13247810}
}
Kostenlos starten →