EUVIMEDEUVIMED entdecken
Open Access

Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?

Małgorzata Pastucha iD, Henryk Skarżyński iD, Krzysztof Kochanek iD, W Wiktor Jedrzejczak iD

Medical Science Monitor · 17.01.2026 · 32

DOI: 10.12659/msm.950916 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Medical Science Monitor, online erschienen am 17.01.2026. Verfasst von Małgorzata Pastucha und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Experimental Audiology, Institute of Physiology and Pathology of Hearing, Warsaw, Poland. Hauptthemen laut MEDLINE: Communication. Die Arbeit wurde bisher 5-mal zitiert (Crossref). Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

BACKGROUND We suggest that testing a large language model (LLM) chatbot in terms of the accuracy of the references it provides could be a powerful, quantifiable means of rating its inherent degree of misinformation, since the accuracy of the bibliographic data can be directly verified. Given the growing reliance on artificial intelligence (AI) tools in academic research and clinical decision-making, such a rating could be extremely useful. MATERIAL AND METHODS In this study, we compared 3 versions of ChatGPT and 3 versions of Gemini by asking them to provide references about 25 highly cited topics in otorhinolaryngology (those with "guidelines" in the title). Answers were sought on 3 consecutive days to assess the variability and consistency of responses. In total, the 6 chatbots returned 1947 references, which were carefully checked against PubMed, Web of Science, and Google Scholar, and rated according to accuracy. Ratings were given based on correct authorship, complete bibliographic details, and proper DOI numbers. RESULTS Common discrepancies noted were wrong author names and erroneous DOI numbers. Across the 6 chatbots, ChatGPT-4.1 (with web search enabled) achieved the best accuracy, with a score of 51%, with Gemini 2.5 Pro being second at 41%. The 2 versions with a web search facility performed better than the 4 versions without. Topics having higher citation counts were associated with lower error rates, suggesting that more widely disseminated scientific findings result in more accurate references. CONCLUSIONS Our findings provide a solid benchmark for rating AI-driven bibliographic retrieval and underline the need for further refinement before these tools can be reliably integrated into academia and clinical applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

CommunicationLarge Language ModelsHumansGenerative Artificial IntelligenceArtificial Intelligence

Autor:innen und Institutionen

  • Department of Experimental Audiology, Institute of Physiology and Pathology of Hearing, Warsaw, Poland.
  • World Hearing Center, Warsaw/Kajetany, Poland.
  • Department of Otorhinolaryngo Surgery, Institute of Physiology and Pathology of Hearing, Warsaw, Poland.

ORCID-Profile: Małgorzata Pastucha, Henryk Skarżyński, Krzysztof Kochanek, W Wiktor Jedrzejczak

Zitieren

APA

Pastucha, M., Skarżyński, H., Kochanek, K., & Jedrzejczak, W. W. (2026). Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?. Medical Science Monitor, 32. https://doi.org/10.12659/msm.950916

Vancouver

Pastucha M, Skarżyński H, Kochanek K, Jedrzejczak WW. Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?. Medical Science Monitor. 2026;32. doi:10.12659/msm.950916

BibTeX
@article{Pastucha2026,
  author = {Pastucha, Małgorzata and Skarżyński, Henryk and Kochanek, Krzysztof and Jedrzejczak, W Wiktor},
  title = {Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?},
  journal = {Medical Science Monitor},
  year = {2026},
  volume = {32},
  doi = {10.12659/msm.950916}
}
Kostenlos starten →