EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Identification of ChatGPT‐Generated Abstracts Within Shoulder and Elbow Surgery Poses a Challenge for Reviewers

Ryan D Stadler, Suleiman Y Sudah, Michael A Moverman, Patrick J Denard, Xavier A Duralde, Grant E Garrigues, Christopher S Klifto, Jonathan C Levy, Surena Namdari, Joaquin Sanchez-Sotelo, Mariano E Menendez

Arthroscopy · 2025

DOI: 10.1016/j.arthro.2024.06.045 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Arthroscopy, 2025. Verfasst von Ryan D Stadler und 10 weiteren Autor:innen. Beteiligte Einrichtung: Rutgers Robert Wood Johnson Medical School, New Brunswick, New Jersey, U.S.A.. Electronic address: ryanstadler23@gmail.com. Hauptthemen laut MEDLINE: Abstracting and Indexing, Artificial Intelligence, Shoulder, Elbow. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

To evaluate the extent to which experienced reviewers can accurately discern between artificial intelligence (AI)–generated and original research abstracts published in the field of shoulder and elbow surgery and compare this with the performance of an AI detection tool.

Methods

Twenty‐five shoulder‐ and elbow‐related articles published in high‐impact journals in 2023 were randomly selected. ChatGPT was prompted with only the abstract title to create an AI‐generated version of each abstract. The resulting 50 abstracts were randomly distributed to and evaluated by 8 blinded peer reviewers with at least 5 years of experience. Reviewers were tasked with distinguishing between original and AI‐generated text. A Likert scale assessed reviewer confidence for each interpretation, and the primary reason guiding assessment of generated text was collected. AI output detector (0%‐100%) and plagiarism (0%‐100%) scores were evaluated using GPTZero.

Results

Reviewers correctly identified 62% of AI‐generated abstracts and misclassified 38% of original abstracts as being AI generated. GPTZero reported a significantly higher probability of AI output among generated abstracts (median, 56%; interquartile range [IQR], 51%‐77%) compared with original abstracts (median, 10%; IQR, 4%‐37%; P < .01). Generated abstracts scored significantly lower on the plagiarism detector (median, 7%; IQR, 5%‐14%) relative to original abstracts (median, 82%; IQR, 72%‐92%; P < .01). Correct identification of AI‐generated abstracts was predominately attributed to the presence of unrealistic data/values. The primary reason for misidentifying original abstracts as AI was attributed to writing style.

Conclusions

Experienced reviewers faced difficulties in distinguishing between human and AI‐generated research content within shoulder and elbow surgery. The presence of unrealistic data facilitated correct identification of AI abstracts, whereas misidentification of original abstracts was often ascribed to writing style. Clinical Relevance With rapidly increasing AI advancements, it is paramount that ethical standards of scientific reporting are upheld. It is therefore helpful to understand the ability of reviewers to identify AI‐generated content.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Abstracting and IndexingArtificial IntelligenceShoulderElbowPeriodicals as TopicHumansGenerative Artificial Intelligence

Autor:innen und Institutionen

  • Rutgers Robert Wood Johnson Medical School, New Brunswick, New Jersey, U.S.A.. Electronic address: ryanstadler23@gmail.com.
  • Department of Orthopaedic Surgery, Monmouth Medical Center, Monmouth, New Jersey, U.S.A.
  • Department of Orthopaedics, University of Utah School of Medicine, Salt Lake City, Utah, U.S.A.
  • Oregon Shoulder Institute, Medford, Oregon, U.S.A.
  • Peachtree Orthopedics, Atlanta, Georgia, U.S.A.
  • Midwest Orthopaedics at Rush University Medical Center, Chicago, Illinois, U.S.A.

Zitieren

APA

Stadler, R. D., Sudah, S. Y., Moverman, M. A., Denard, P. J., Duralde, X. A., Garrigues, G. E., Klifto, C. S., Levy, J. C., Namdari, S., Sanchez-Sotelo, J., & Menendez, M. E. (2025). Identification of ChatGPT‐Generated Abstracts Within Shoulder and Elbow Surgery Poses a Challenge for Reviewers. Arthroscopy. https://doi.org/10.1016/j.arthro.2024.06.045

Vancouver

Stadler RD, Sudah SY, Moverman MA, Denard PJ, Duralde XA, Garrigues GE, et al. Identification of ChatGPT‐Generated Abstracts Within Shoulder and Elbow Surgery Poses a Challenge for Reviewers. Arthroscopy. 2025. doi:10.1016/j.arthro.2024.06.045

BibTeX
@article{Stadler2025,
  author = {Stadler, Ryan D and Sudah, Suleiman Y and Moverman, Michael A and Denard, Patrick J and Duralde, Xavier A and Garrigues, Grant E and Klifto, Christopher S and Levy, Jonathan C and Namdari, Surena and Sanchez-Sotelo, Joaquin and Menendez, Mariano E},
  title = {Identification of ChatGPT‐Generated Abstracts Within Shoulder and Elbow Surgery Poses a Challenge for Reviewers},
  journal = {Arthroscopy},
  year = {2025},
  doi = {10.1016/j.arthro.2024.06.045}
}
Kostenlos starten →