EUVIMEDEUVIMED entdecken
Open Access

Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis

Hassan Tarek Hakam iD, Robert Prill iD, Lisa Korte iD, Bruno Lovreković iD, Marko Ostojić iD, Nikolai Ramadanov iD, Felix Muehlensiepen iD

JMIR Formative Research · 16.02.2024 · 8 · e52164

DOI: 10.2196/52164 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 16.02.2024. Verfasst von Hassan Tarek Hakam und 6 weiteren Autor:innen. Beteiligte Einrichtung: Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

As large language models (LLMs) are becoming increasingly integrated into different aspects of health care, questions about the implications for medical academic literature have begun to emerge. Key aspects such as authenticity in academic writing are at stake with artificial intelligence (AI) generating highly linguistically accurate and grammatically sound texts.

Objective

The objective of this study is to compare human-written with AI-generated scientific literature in orthopedics and sports medicine.

Methods

Five original abstracts were selected from the PubMed database. These abstracts were subsequently rewritten with the assistance of 2 LLMs with different degrees of proficiency. Subsequently, researchers with varying degrees of expertise and with different areas of specialization were asked to rank the abstracts according to linguistic and methodological parameters. Finally, researchers had to classify the articles as AI generated or human written.

Results

Neither the researchers nor the AI-detection software could successfully identify the AI-generated texts. Furthermore, the criteria previously suggested in the literature did not correlate with whether the researchers deemed a text to be AI generated or whether they judged the article correctly based on these parameters.

Conclusions

The primary finding of this study was that researchers were unable to distinguish between LLM-generated and human-written texts. However, due to the small sample size, it is not possible to generalize the results of this study. As is the case with any tool used in academic research, the potential to cause harm can be mitigated by relying on the transparency and integrity of the researchers. With scientific integrity at stake, further research with a similar study design should be conducted to determine the magnitude of this issue.

Schlagwörter

Stichwörter der Autor:innen

AILLMartificial intelligencedetectionfeedbacklarge language modelmedical databaseorthopedicorthopedic surgeryorthopedicsqualitative studyresearchscientific integritysports medicinestudy design

Autor:innen und Institutionen

  • Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany.
  • Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany.
  • Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany.
  • Center of Health Services Research, Faculty of Health Sciences, University Clinic of Brandenburg, Rüdersdorf bei Berlin, Germany.
  • Faculty of Orthopaedics, University Hospital Merkur, Zagreb, Croatia.
  • Departement of Orthopaedics, University Hospital Mostar, Mostar, Bosnia and Herzegovina.

ORCID-Profile: Hassan Tarek Hakam, Robert Prill, Lisa Korte, Bruno Lovreković, Marko Ostojić, Nikolai Ramadanov, Felix Muehlensiepen

Zitieren

APA

Hakam, H. T., Prill, R., Korte, L., Lovreković, B., Ostojić, M., Ramadanov, N., & Muehlensiepen, F. (2024). Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research, 8, e52164. https://doi.org/10.2196/52164

Vancouver

Hakam HT, Prill R, Korte L, Lovreković B, Ostojić M, Ramadanov N, et al. Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research. 2024;8:e52164. doi:10.2196/52164

BibTeX
@article{Hakam2024,
  author = {Hakam, Hassan Tarek and Prill, Robert and Korte, Lisa and Lovreković, Bruno and Ostojić, Marko and Ramadanov, Nikolai and Muehlensiepen, Felix},
  title = {Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e52164},
  doi = {10.2196/52164}
}
Kostenlos starten →