EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study

Shu-Li Cheng iD, Shih-Jen Tsai iD, Ya-Mei Bai iD, Chih-Hung Ko iD, Chih-Wei Hsu iD, Fu-Chi Yang iD, Chia-Kuang Tsai iD, Yu-Kang Tu iD, Szu-Nian Yang iD, Ping-Tao Tseng iD, Tien-Wei Hsu iD, Chih-Sung Liang iD und 1 weitere

Journal of Medical Internet Research · 2023

DOI: 10.2196/51229 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift Journal of Medical Internet Research, 2023. Verfasst von Shu-Li Cheng und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Nursing, Mackay Medical College, Taipei, Taiwan. Hauptthemen laut MEDLINE: Artificial Intelligence, Research. Die Arbeit wurde bisher 56-mal zitiert (Crossref) und nennt 11 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

ChatGPT may act as a research assistant to help organize the direction of thinking and summarize research findings. However, few studies have examined the quality, similarity (abstracts being similar to the original one), and accuracy of the abstracts generated by ChatGPT when researchers provide full-text basic research papers.

Objective

We aimed to assess the applicability of an artificial intelligence (AI) model in generating abstracts for basic preclinical research.

Methods

We selected 30 basic research papers from Nature, Genome Biology, and Biological Psychiatry. Excluding abstracts, we inputted the full text into ChatPDF, an application of a language model based on ChatGPT, and we prompted it to generate abstracts with the same style as used in the original papers. A total of 8 experts were invited to evaluate the quality of these abstracts (based on a Likert scale of 0-10) and identify which abstracts were generated by ChatPDF, using a blind approach. These abstracts were also evaluated for their similarity to the original abstracts and the accuracy of the AI content.

Results

The quality of ChatGPT-generated abstracts was lower than that of the actual abstracts (10-point Likert scale: mean 4.72, SD 2.09 vs mean 8.09, SD 1.03; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceResearchHumansCross-Sectional StudiesResearch PersonnelLanguage

Stichwörter der Autor:innen

AI-generated scientific contentChatGPTLLMNLPabstractabstractsacademic researchartificial intelligenceextractextractiongenerationgenerativelanguage modellanguage modelsnatural language processing

Autor:innen und Institutionen

  • Department of Nursing, Mackay Medical College, Taipei, Taiwan.
  • Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan.
  • Division of Psychiatry, School of Medicine, National Yang-Ming University, Taipei, Taiwan.
  • Department of Psychiatry, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan.
  • Department of Psychiatry, College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan.
  • Department of Psychiatry, Kaohsiung Municipal Siaogang Hospital, Kaohsiung Medical University, Kaohsiung, Taiwan.

ORCID-Profile: Shu-Li Cheng, Shih-Jen Tsai, Ya-Mei Bai, Chih-Hung Ko, Chih-Wei Hsu, Fu-Chi Yang, Chia-Kuang Tsai, Yu-Kang Tu, Szu-Nian Yang, Ping-Tao Tseng, Tien-Wei Hsu, Chih-Sung Liang, Kuan-Pin Su

Zitieren

APA

Cheng, S. L., Tsai, S. J., Bai, Y. M., Ko, C. H., Hsu, C. W., Yang, F. C., Tsai, C. K., Tu, Y. K., Yang, S. N., Tseng, P. T., Hsu, T. W., Liang, C. S., & Su, K. P. (2023). Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study. Journal of Medical Internet Research. https://doi.org/10.2196/51229

Vancouver

Cheng SL, Tsai SJ, Bai YM, Ko CH, Hsu CW, Yang FC, et al. Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study. Journal of Medical Internet Research. 2023. doi:10.2196/51229

BibTeX
@article{Cheng2023,
  author = {Cheng, Shu-Li and Tsai, Shih-Jen and Bai, Ya-Mei and Ko, Chih-Hung and Hsu, Chih-Wei and Yang, Fu-Chi and Tsai, Chia-Kuang and Tu, Yu-Kang and Yang, Szu-Nian and Tseng, Ping-Tao and Hsu, Tien-Wei and Liang, Chih-Sung and Su, Kuan-Pin},
  title = {Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  doi = {10.2196/51229}
}
Kostenlos starten →