EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis

Hong Jin Kim iD, Jae Hyuk Yang iD, Dong-Gune Chang iD, Lawrence G Lenke iD, Javier Pizones iD, René Castelein iD, Kota Watanabe iD, Per D Trobisch iD, Gregory M Mundis iD, Seung Woo Suh iD, Se-Il Suk iD

Journal of Medical Internet Research · 26.06.2024 · 26 · e52001

DOI: 10.2196/52001 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 26.06.2024. Verfasst von Hong Jin Kim und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedic Surgery, Inje University Sanggye Paik Hospital, College of Medicine, Inje University, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Spine, Abstracting and Indexing. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Due to recent advances in artificial intelligence (AI), language model applications can generate logical text output that is difficult to distinguish from human writing. ChatGPT (OpenAI) and Bard (subsequently rebranded as “Gemini”; Google AI) were developed using distinct approaches, but little has been studied about the difference in their capability to generate the abstract. The use of AI to write scientific abstracts in the field of spine surgery is the center of much debate and controversy.

Objective

The objective of this study is to assess the reproducibility of the structured abstracts generated by ChatGPT and Bard compared to human-written abstracts in the field of spine surgery.

Methods

In total, 60 abstracts dealing with spine sections were randomly selected from 7 reputable journals and used as ChatGPT and Bard input statements to generate abstracts based on supplied paper titles. A total of 174 abstracts, divided into human-written abstracts, ChatGPT-generated abstracts, and Bard-generated abstracts, were evaluated for compliance with the structured format of journal guidelines and consistency of content. The likelihood of plagiarism and AI output was assessed using the iThenticate and ZeroGPT programs, respectively. A total of 8 reviewers in the spinal field evaluated 30 randomly extracted abstracts to determine whether they were produced by AI or human authors.

Results

The proportion of abstracts that met journal formatting guidelines was greater among ChatGPT abstracts (34/60, 56.6%) compared with those generated by Bard (6/54, 11.1%; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SpineAbstracting and IndexingHumansReproducibility of ResultsArtificial IntelligenceWriting

Stichwörter der Autor:innen

AIBardChatGPTabstractartificial intelligencechatbotethicsformatting guidelinesjournal guidelineslanguage modelorthopedic surgeryplagiarismscientific abstractspinespine surgery

Autor:innen und Institutionen

  • Department of Orthopedic Surgery, Inje University Sanggye Paik Hospital, College of Medicine, Inje University, Seoul, Republic of Korea.
  • Department of Orthopedic Surgery, Korea University Anam Hospital, College of Medicine, Korea University, Seoul, Republic of Korea.
  • Department of Orthopedic Surgery, The Daniel and Jane Och Spine Hospital, Columbia University, New York, NY, United States.
  • Department of Orthopedic Surgery, Hospital Universitario La Paz, Madrid, Spain.
  • Department of Orthopedic Surgery, University Medical Centre Utrecht, Utrecht, Netherlands.
  • Department of Orthopedic Surgery, Keio University School of Medicine, Tokyo, Japan.

ORCID-Profile: Hong Jin Kim, Jae Hyuk Yang, Dong-Gune Chang, Lawrence G Lenke, Javier Pizones, René Castelein, Kota Watanabe, Per D Trobisch, Gregory M Mundis, Seung Woo Suh, Se-Il Suk

Zitieren

APA

Kim, H. J., Yang, J. H., Chang, D. G., Lenke, L. G., Pizones, J., Castelein, R., Watanabe, K., Trobisch, P. D., Mundis, G. M., Suh, S. W., & Suk, S. I. (2024). Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis. Journal of Medical Internet Research, 26, e52001. https://doi.org/10.2196/52001

Vancouver

Kim HJ, Yang JH, Chang DG, Lenke LG, Pizones J, Castelein R, et al. Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis. Journal of Medical Internet Research. 2024;26:e52001. doi:10.2196/52001

BibTeX
@article{Kim2024,
  author = {Kim, Hong Jin and Yang, Jae Hyuk and Chang, Dong-Gune and Lenke, Lawrence G and Pizones, Javier and Castelein, René and Watanabe, Kota and Trobisch, Per D and Mundis, Gregory M and Suh, Seung Woo and Suk, Se-Il},
  title = {Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e52001},
  doi = {10.2196/52001}
}
Kostenlos starten →