Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?
Learned Publishing · 2025-10 · 38 (4)
DOI: 10.1002/leap.2018 ↗Kurz gefasst
Fachartikel in der Zeitschrift Learned Publishing, online erschienen am 04.08.2025. Verfasst von Mike Thelwall und 4 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) like ChatGPT seem to be increasingly used for information seeking and analysis, including to support academic literature reviews. To test whether the results might sometimes include retracted research, we identified 217 retracted or otherwise concerning academic studies with high altmetric scores and asked ChatGPT 4o‐mini to evaluate their quality 30 times each. Surprisingly, none of its 6510 reports mentioned that the articles were retracted or had relevant errors, and it gave 190 relatively high scores (world leading, internationally excellent, or close). The 27 articles with the lowest scores were mostly accused of being weak, although the topic (but not the article) was described as controversial in five cases (e.g., about hydroxychloroquine for COVID‐19). In a follow‐up investigation, 61 claims were extracted from retracted articles from the set, and ChatGPT 4o‐mini was asked 10 times whether each was true. It gave a definitive yes or a positive response two‐thirds of the time, including for at least one statement that had been shown to be false over a decade ago. The results therefore emphasise, from an academic knowledge perspective, the importance of verifying information from LLMs when using them for information seeking or analysis.
Zitieren
APA
Thelwall, M., Lehtisaari, M., Katsirea, I., Holmberg, K., & Zheng, E. (2025). Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?. Learned Publishing, 38(4). https://doi.org/10.1002/leap.2018
Vancouver
Thelwall M, Lehtisaari M, Katsirea I, Holmberg K, Zheng E. Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?. Learned Publishing. 2025;38(4). doi:10.1002/leap.2018
BibTeX
@article{Thelwall2025,
author = {Thelwall, Mike and Lehtisaari, Marianna and Katsirea, Irini and Holmberg, Kim and Zheng, Er‐Te},
title = {Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?},
journal = {Learned Publishing},
year = {2025},
volume = {38},
number = {4},
doi = {10.1002/leap.2018}
}
EUVIMED