EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitOpen Access

Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest

Gisele S Silva iD, Rohan Khera iD, Lee H Schwamm iD

Stroke · 2024-10 · 55 (10) · 2573-2578

DOI: 10.1161/strokeaha.124.045012 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift Stroke, 2024-10. Verfasst von Gisele S Silva und 2 weiteren Autor:innen. Beteiligte Einrichtung: Hospital Israelita Albert Einstein and Departamento de Neurologia e Neurocirurgia, Universidade Federal de São Paulo, Brazil (G.S.S.). Hauptthemen laut MEDLINE: Stroke, Artificial Intelligence. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHLBI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial intelligence (AI) large language models (LLMs) now produce human-like general text and images. LLMs’ ability to generate persuasive scientific essays that undergo evaluation under traditional peer review has not been systematically studied. To measure perceptions of quality and the nature of authorship, we conducted a competitive essay contest in 2024 with both human and AI participants. Human authors and 4 distinct LLMs generated essays on controversial topics in stroke care and outcomes research. A panel of Stroke Editorial Board members (mostly vascular neurologists), blinded to author identity and with varying levels of AI expertise, rated the essays for quality, persuasiveness, best in topic, and author type. Among 34 submissions (22 human and 12 LLM) scored by 38 reviewers, human and AI essays received mostly similar ratings, though AI essays were rated higher for composition quality. Author type was accurately identified only 50% of the time, with prior LLM experience associated with improved accuracy. In multivariable analyses adjusted for author attributes and essay quality, only persuasiveness was independently associated with odds of a reviewer assigning AI as author type (adjusted odds ratio, 1.53 [95% CI, 1.09–2.16]; P =0.01). In conclusion, a group of experienced editorial board members struggled to distinguish human versus AI authorship, with a bias against best in topic for essays judged to be AI generated. Scientific journals may benefit from educating reviewers on the types and uses of AI in scientific writing and developing thoughtful policies on the appropriate use of AI in authoring manuscripts.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

StrokeArtificial IntelligenceHumansAuthorshipPeriodicals as TopicPeer Review, Research

Stichwörter der Autor:innen

artificial intelligenceneurologistspeer reviewstrokewriting

Autor:innen und Institutionen

  • Hospital Israelita Albert Einstein and Departamento de Neurologia e Neurocirurgia, Universidade Federal de São Paulo, Brazil (G.S.S.).
  • Section of Cardiovascular Medicine (R.K.), Yale School of Medicine, New Haven, CT.
  • Biomedical Informatics and Data Science (R.K., L.H.S.), Yale School of Medicine, New Haven, CT.
  • Section of Health Informatics, Department of Biostatistics, Yale School of Public Health, New Haven, CT (R.K.).
  • Department of Neurology (L.H.S.), Yale School of Medicine, New Haven, CT.
  • Digital and Technology Solutions, Yale New Haven Health System, CT (L.H.S.).

ORCID-Profile: Gisele S Silva, Rohan Khera, Lee H Schwamm

Förderung

  • NHLBI NIH HHS

Zitieren

APA

Silva, G. S., Khera, R., & Schwamm, L. H. (2024). Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest. Stroke, 55(10), 2573-2578. https://doi.org/10.1161/strokeaha.124.045012

Vancouver

Silva GS, Khera R, Schwamm LH. Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest. Stroke. 2024;55(10):2573-2578. doi:10.1161/strokeaha.124.045012

BibTeX
@article{Silva2024,
  author = {Silva, Gisele S and Khera, Rohan and Schwamm, Lee H},
  title = {Reviewer Experience Detecting and Judging Human Versus Artificial Intelligence Content: The Stroke Journal Essay Contest},
  journal = {Stroke},
  year = {2024},
  volume = {55},
  number = {10},
  pages = {2573--2578},
  doi = {10.1161/strokeaha.124.045012}
}
Kostenlos starten →