Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study
JMIR Formative Research · 03.11.2026 · 10 · e103592-e103592
DOI: 10.2196/103592 ↗Kurz gefasst
Beobachtungsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 03.11.2026. Verfasst von Francesco De Caro und 12 weiteren Autor:innen. Beteiligte Einrichtung: Public Health Laboratory for the Analysis of Community Health Needs, Department of Medicine and Surgery, University of Salerno, Baronissi, Campania, …. Hauptthemen laut MEDLINE: Biomedical Research. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The use of generative AI in research and clinical practice is rapidly expanding, offering unprecedented capabilities in pattern extraction and data analysis. However, its application raises critical concerns regarding the accuracy and reliability of the information provided.
Objective
The aim of this study is to analyze and compare the performance of 4 generative AI platforms (ChatGPT, Grok, Meta AI, and Perplexity) in retrieving bibliographic references within the medical and health care field and to assess factual accuracy and information retrieval strategies.
Methods
A cross-sectional observational study was conducted based on the simulation of 3 bibliographic queries with varying levels of specificity (aflatoxin toxicity, hospital sanitization, and human papillomavirus vaccination). Each prompt was administered in triple iteration on each platform operating in “Guest” mode. Factual accuracy was verified through a manual cross-check against the Scopus and Google Scholar databases using a hierarchical query strategy prioritized by DOI. The incidence of hallucinations was quantified via a standardized scoring system (range 0‐5) based on metadata correctness. Categorical associations were analyzed using the Pearson chi-square test, while nonparametric continuous variables (hallucination score, publication year, and citation count) were compared using the Kruskal-Wallis test.
Results
Of the 360 requested references, 273 were generated (task completion rate: 75.8%). Among these, 190 adhered to the requested article format (instruction adherence rate: 69.6% of generated outputs). Fact-checking revealed that 128 of these 190 validly formatted articles (metadata accuracy rate: 67.4%) showed an exact match with real evidence, resulting in an overall strict yield of 35.6% (128/360) from the initial requested references. The ability to provide real references depended significantly on the AI model used (χ 6 ²=192.5; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Public Health Laboratory for the Analysis of Community Health Needs, Department of Medicine and Surgery, University of Salerno, Baronissi, Campania, Italy.
- Department of Medicine and Surgery, University of Salerno, Baronissi Campus, Baronissi, Campania, Italy, 39 3385430632.
- Department of Political and Social Studies, University of Salerno, Fisciano, Campania, Italy.
Zitieren
APA
De Caro, F., Della Rocca, F., Mea, A., Tomeo, M., Caruccio, S., Baldi, M., D'Arienzo, G., De Caro, R., Santoro, E., Cersosimo, G., Boccia, G., Capunzo, M., & Moccia, G. (2026). Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study. JMIR Formative Research, 10, e103592-e103592. https://doi.org/10.2196/103592
Vancouver
De Caro F, Della Rocca F, Mea A, Tomeo M, Caruccio S, Baldi M, et al. Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study. JMIR Formative Research. 2026;10:e103592-e103592. doi:10.2196/103592
BibTeX
@article{DeCaro2026,
author = {De Caro, Francesco and Della Rocca, Federico and Mea, Alessandra and Tomeo, Matteo and Caruccio, Simona and Baldi, Myriam and D'Arienzo, Giuseppe and De Caro, Raffaele and Santoro, Emanuela and Cersosimo, Giuseppina and Boccia, Giovanni and Capunzo, Mario and Moccia, Giuseppina},
title = {Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study},
journal = {JMIR Formative Research},
year = {2026},
volume = {10},
pages = {e103592--e103592},
doi = {10.2196/103592}
}
EUVIMED