Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest
Stroke · 2024-10 · 55 (10) · 2573-2578
DOI: 10.1161/strokeaha.124.045012 ↗At a glance
Übersichtsarbeit in Stroke, 2024-10. Written by Gisele S Silva and 2 co-authors. Institution: Hospital Israelita Albert Einstein and Departamento de Neurologia e Neurocirurgia, Universidade Federal de São Paulo, Brazil (G.S.S.). Major MEDLINE topics: Stroke, Artificial Intelligence. Cited 18 times so far (Crossref); 16 references. The full text is freely available in PubMed Central. Funded by: NHLBI NIH HHS.
Abstract (original language)
Artificial intelligence (AI) large language models (LLMs) now produce human-like general text and images. LLMs’ ability to generate persuasive scientific essays that undergo evaluation under traditional peer review has not been systematically studied. To measure perceptions of quality and the nature of authorship, we conducted a competitive essay contest in 2024 with both human and AI participants. Human authors and 4 distinct LLMs generated essays on controversial topics in stroke care and outcomes research. A panel of Stroke Editorial Board members (mostly vascular neurologists), blinded to author identity and with varying levels of AI expertise, rated the essays for quality, persuasiveness, best in topic, and author type. Among 34 submissions (22 human and 12 LLM) scored by 38 reviewers, human and AI essays received mostly similar ratings, though AI essays were rated higher for composition quality. Author type was accurately identified only 50% of the time, with prior LLM experience associated with improved accuracy. In multivariable analyses adjusted for author attributes and essay quality, only persuasiveness was independently associated with odds of a reviewer assigning AI as author type (adjusted odds ratio, 1.53 [95% CI, 1.09–2.16]; P =0.01). In conclusion, a group of experienced editorial board members struggled to distinguish human versus AI authorship, with a bias against best in topic for essays judged to be AI generated. Scientific journals may benefit from educating reviewers on the types and uses of AI in scientific writing and developing thoughtful policies on the appropriate use of AI in authoring manuscripts.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Hospital Israelita Albert Einstein and Departamento de Neurologia e Neurocirurgia, Universidade Federal de São Paulo, Brazil (G.S.S.).
- Section of Cardiovascular Medicine (R.K.), Yale School of Medicine, New Haven, CT.
- Biomedical Informatics and Data Science (R.K., L.H.S.), Yale School of Medicine, New Haven, CT.
- Section of Health Informatics, Department of Biostatistics, Yale School of Public Health, New Haven, CT (R.K.).
- Department of Neurology (L.H.S.), Yale School of Medicine, New Haven, CT.
- Digital and Technology Solutions, Yale New Haven Health System, CT (L.H.S.).
ORCID profiles: Gisele S Silva, Rohan Khera, Lee H Schwamm
Funding
- NHLBI NIH HHS
Cite
APA
Silva, G. S., Khera, R., & Schwamm, L. H. (2024). Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest. Stroke, 55(10), 2573-2578. https://doi.org/10.1161/strokeaha.124.045012
Vancouver
Silva GS, Khera R, Schwamm LH. Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest. Stroke. 2024;55(10):2573-2578. doi:10.1161/strokeaha.124.045012
BibTeX
@article{Silva2024,
author = {Silva, Gisele S and Khera, Rohan and Schwamm, Lee H},
title = {Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest},
journal = {Stroke},
year = {2024},
volume = {55},
number = {10},
pages = {2573--2578},
doi = {10.1161/strokeaha.124.045012}
}
EUVIMED