Evaluating Web Retrieval–Assisted Large Language Models With and Without Whitelisting for Evidence-Based Neurology: Comparative Study
Journal of Medical Internet Research · 2025
DOI: 10.2196/79379 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Lars Masanneck und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Dusseldorf, Germany. Hauptthemen laut MEDLINE: Neurology, Internet, Information Storage and Retrieval, Evidence-Based Medicine. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) coupled with real-time web retrieval are reshaping how clinicians and patients locate medical evidence, and as major search providers fuse LLMs into their interfaces, this hybrid approach might become the new “gateway” to the internet. However, open-web retrieval exposes models to nonprofessional sources, risking hallucinations and factual errors that might jeopardize evidence-based care.
Objective
We aimed to quantify the impact of guideline-domain whitelisting on the answer quality of 3 publicly available Perplexity web-based retrieval-augmented generation (RAG) models and compare their performance using a purpose-built, biomedical literature RAG system (OpenEvidence).
Methods
We applied a validated 130-item question set derived from the American Academy of Neurology (AAN) guidelines (65 factual and 65 case based). Perplexity Sonar, Sonar-Pro, and Sonar-Reasoning-Pro were each queried 4 times per question with open-web retrieval and again with retrieval restricted to aan.com and neurology.org (“whitelisted”). OpenEvidence was queried 4 times. Two neurologists, blinded to condition, scored each response (0=wrong, 1=inaccurate, and 2=correct); any disagreements that arose were resolved by a third neurologist. Ordinal logistic models were used to assess the influence of question type and source category (AAN or neurology vs nonprofessional) on accuracy.
Results
From the 3640 LLM answers that were rated (interrater agreement: κ=0.86), correct-answer rates were as follows (open vs whitelisted, respectively): Sonar, 60% vs 78%, Sonar-Pro, 80% vs 88%, and Sonar-Reasoning-Pro, 81% vs 89%; for OpenEvidence, the correct-answer rate was 82%. A Friedman test on modal scores across the 7 configurations was significant (χ26=73.7; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Neurology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Dusseldorf, Germany.
ORCID-Profile: Lars Masanneck, Paula Zoe Epping, Sven G Meuth, Marc Pawlitzki
Zitieren
APA
Masanneck, L., Epping, P. Z., Meuth, S. G., & Pawlitzki, M. (2025). Evaluating Web Retrieval–Assisted Large Language Models With and Without Whitelisting for Evidence-Based Neurology: Comparative Study. Journal of Medical Internet Research. https://doi.org/10.2196/79379
Vancouver
Masanneck L, Epping PZ, Meuth SG, Pawlitzki M. Evaluating Web Retrieval–Assisted Large Language Models With and Without Whitelisting for Evidence-Based Neurology: Comparative Study. Journal of Medical Internet Research. 2025. doi:10.2196/79379
BibTeX
@article{Masanneck2025,
author = {Masanneck, Lars and Epping, Paula Zoe and Meuth, Sven G and Pawlitzki, Marc},
title = {Evaluating Web Retrieval–Assisted Large Language Models With and Without Whitelisting for Evidence-Based Neurology: Comparative Study},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/79379}
}
EUVIMED