EUVIMEDPoznaj EUVIMED
VergleichsstudieOpen access

Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis

Hong Jin Kim iD, Jae Hyuk Yang iD, Dong-Gune Chang iD, Lawrence G Lenke iD, Javier Pizones iD, René Castelein iD, Kota Watanabe iD, Per D Trobisch iD, Gregory M Mundis iD, Seung Woo Suh iD, Se-Il Suk iD

Journal of Medical Internet Research · 2024-06-26 · 26 · e52001

DOI: 10.2196/52001 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2024-06-26. Written by Hong Jin Kim and 10 co-authors. Institution: Department of Orthopedic Surgery, Inje University Sanggye Paik Hospital, College of Medicine, Inje University, Seoul, Republic of Korea. Major MEDLINE topics: Spine, Abstracting and Indexing. Cited 22 times so far (Crossref); 24 references. The full text is freely available in PubMed Central.

AbstraktSubjectsDane bibliograficzneCiteRelated studies

Abstract (original language)

Background

Due to recent advances in artificial intelligence (AI), language model applications can generate logical text output that is difficult to distinguish from human writing. ChatGPT (OpenAI) and Bard (subsequently rebranded as “Gemini”; Google AI) were developed using distinct approaches, but little has been studied about the difference in their capability to generate the abstract. The use of AI to write scientific abstracts in the field of spine surgery is the center of much debate and controversy.

Objective

The objective of this study is to assess the reproducibility of the structured abstracts generated by ChatGPT and Bard compared to human-written abstracts in the field of spine surgery.

Methods

In total, 60 abstracts dealing with spine sections were randomly selected from 7 reputable journals and used as ChatGPT and Bard input statements to generate abstracts based on supplied paper titles. A total of 174 abstracts, divided into human-written abstracts, ChatGPT-generated abstracts, and Bard-generated abstracts, were evaluated for compliance with the structured format of journal guidelines and consistency of content. The likelihood of plagiarism and AI output was assessed using the iThenticate and ZeroGPT programs, respectively. A total of 8 reviewers in the spinal field evaluated 30 randomly extracted abstracts to determine whether they were produced by AI or human authors.

Results

The proportion of abstracts that met journal formatting guidelines was greater among ChatGPT abstracts (34/60, 56.6%) compared with those generated by Bard (6/54, 11.1%; P

Subjects

MeSH terms (MEDLINE)

SpineAbstracting and IndexingHumansReproducibility of ResultsArtificial IntelligenceWriting

Author keywords

AIBardChatGPTabstractartificial intelligencechatbotethicsformatting guidelinesjournal guidelineslanguage modelorthopedic surgeryplagiarismscientific abstractspinespine surgery

Authors and institutions

  • Department of Orthopedic Surgery, Inje University Sanggye Paik Hospital, College of Medicine, Inje University, Seoul, Republic of Korea.
  • Department of Orthopedic Surgery, Korea University Anam Hospital, College of Medicine, Korea University, Seoul, Republic of Korea.
  • Department of Orthopedic Surgery, The Daniel and Jane Och Spine Hospital, Columbia University, New York, NY, United States.
  • Department of Orthopedic Surgery, Hospital Universitario La Paz, Madrid, Spain.
  • Department of Orthopedic Surgery, University Medical Centre Utrecht, Utrecht, Netherlands.
  • Department of Orthopedic Surgery, Keio University School of Medicine, Tokyo, Japan.

ORCID profiles: Hong Jin Kim, Jae Hyuk Yang, Dong-Gune Chang, Lawrence G Lenke, Javier Pizones, René Castelein, Kota Watanabe, Per D Trobisch, Gregory M Mundis, Seung Woo Suh, Se-Il Suk

Cite

APA

Kim, H. J., Yang, J. H., Chang, D. G., Lenke, L. G., Pizones, J., Castelein, R., Watanabe, K., Trobisch, P. D., Mundis, G. M., Suh, S. W., & Suk, S. I. (2024). Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis. Journal of Medical Internet Research, 26, e52001. https://doi.org/10.2196/52001

Vancouver

Kim HJ, Yang JH, Chang DG, Lenke LG, Pizones J, Castelein R, et al. Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis. Journal of Medical Internet Research. 2024;26:e52001. doi:10.2196/52001

BibTeX
@article{Kim2024,
  author = {Kim, Hong Jin and Yang, Jae Hyuk and Chang, Dong-Gune and Lenke, Lawrence G and Pizones, Javier and Castelein, René and Watanabe, Kota and Trobisch, Per D and Mundis, Gregory M and Suh, Seung Woo and Suk, Se-Il},
  title = {Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e52001},
  doi = {10.2196/52001}
}
Start for free →