EUVIMEDEUVIMED entdecken
Open Access

Do Large Language Models Link to Incorrect Citations? A PubMed and DOI Benchmark

Goktug Ozkan iD

Applied Clinical Informatics · 01.08.2026 · 17 (04) · 950-954

DOI: 10.1055/a-2972-4974 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.08.2026. Verfasst von Goktug Ozkan. Beteiligte Einrichtung: Maltepe State Hospital, Department of Internal Medicine, Istanbul, Türkiye, Istanbul. Hauptthemen laut MEDLINE: Large Language Models, PubMed, Benchmarking. Die Arbeit nennt 8 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models can return citations whose PubMed Identifier (PMID), digital object identifier (DOI), title, and journal appear plausible but identify different papers. Checking one valid identifier may therefore fail to detect an incorrect citation bundle.

Objective

This study aimed to measure citation metadata incoherence in responses from five large language model snapshots to an internal medicine prompt benchmark.

Methods

Five models answered 30 synthetic prompts each, producing 150 model-by-prompt responses. Twenty-six prompts per model requested citations, and four were designed to elicit abstention. A corrected parser extracted 300 citation units. We checked reported PMIDs against PubMed metadata, queried reported DOIs separately, and rechecked DOI strings not found in PubMed through Crossref. Response-level outcomes were primary because citation units were nested within responses. The author reviewed a deterministic stratified sample of 50 units for parser capture and citation identity.

Results

Among 130 citation-expected responses, 69 contained a DOI that resolved to a different PMID than the PMID supplied by the model, 100 contained at least one automated metadata mismatch, and 28 contained no citation. Those 100 were 69 DOI–PMID, 29 other, and 2 unresolvable. All 20 abstention trap responses abstained. Among 116 units whose reported DOI resolved in PubMed, 99 resolved to a different PMID and 17 to the reported PMID. Human review found correct parser capture in 48 of 50 sampled units. Two section boundary errors were corrected before the final rerun; the primary counts did not change. Human review classified 32 sampled units as incoherent, 7 as coherent, 6 as having an unresolvable PMID, and 5 as containing no citation. Clinical claim support was not assessed.

Conclusion

Citation identity failures occurred in all five tested model snapshots. Evaluation and clinical evidence workflows should confirm that the PMID, DOI, title, and journal identify the same source before treating a model-supplied citation as evidence.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPubMedBenchmarkingMetadata

Autor:innen und Institutionen

  • Maltepe State Hospital, Department of Internal Medicine, Istanbul, Türkiye, Istanbul.

ORCID-Profile: Goktug Ozkan

Zitieren

APA

Ozkan, G. (2026). Do Large Language Models Link to Incorrect Citations? A PubMed and DOI Benchmark. Applied Clinical Informatics, 17(04), 950-954. https://doi.org/10.1055/a-2972-4974

Vancouver

Ozkan G. Do Large Language Models Link to Incorrect Citations? A PubMed and DOI Benchmark. Applied Clinical Informatics. 2026;17(04):950-954. doi:10.1055/a-2972-4974

BibTeX
@article{Ozkan2026,
  author = {Ozkan, Goktug},
  title = {Do Large Language Models Link to Incorrect Citations? A PubMed and DOI Benchmark},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {04},
  pages = {950--954},
  doi = {10.1055/a-2972-4974}
}
Kostenlos starten →