Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis
JMIR Formative Research · 16.02.2024 · 8 · e52164
DOI: 10.2196/52164 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 16.02.2024. Verfasst von Hassan Tarek Hakam und 6 weiteren Autor:innen. Beteiligte Einrichtung: Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
As large language models (LLMs) are becoming increasingly integrated into different aspects of health care, questions about the implications for medical academic literature have begun to emerge. Key aspects such as authenticity in academic writing are at stake with artificial intelligence (AI) generating highly linguistically accurate and grammatically sound texts.
Objective
The objective of this study is to compare human-written with AI-generated scientific literature in orthopedics and sports medicine.
Methods
Five original abstracts were selected from the PubMed database. These abstracts were subsequently rewritten with the assistance of 2 LLMs with different degrees of proficiency. Subsequently, researchers with varying degrees of expertise and with different areas of specialization were asked to rank the abstracts according to linguistic and methodological parameters. Finally, researchers had to classify the articles as AI generated or human written.
Results
Neither the researchers nor the AI-detection software could successfully identify the AI-generated texts. Furthermore, the criteria previously suggested in the literature did not correlate with whether the researchers deemed a text to be AI generated or whether they judged the article correctly based on these parameters.
Conclusions
The primary finding of this study was that researchers were unable to distinguish between LLM-generated and human-written texts. However, due to the small sample size, it is not possible to generalize the results of this study. As is the case with any tool used in academic research, the potential to cause harm can be mitigated by relying on the transparency and integrity of the researchers. With scientific integrity at stake, further research with a similar study design should be conducted to determine the magnitude of this issue.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany.
- Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany.
- Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany.
- Center of Health Services Research, Faculty of Health Sciences, University Clinic of Brandenburg, Rüdersdorf bei Berlin, Germany.
- Faculty of Orthopaedics, University Hospital Merkur, Zagreb, Croatia.
- Departement of Orthopaedics, University Hospital Mostar, Mostar, Bosnia and Herzegovina.
ORCID-Profile: Hassan Tarek Hakam, Robert Prill, Lisa Korte, Bruno Lovreković, Marko Ostojić, Nikolai Ramadanov, Felix Muehlensiepen
Zitieren
APA
Hakam, H. T., Prill, R., Korte, L., Lovreković, B., Ostojić, M., Ramadanov, N., & Muehlensiepen, F. (2024). Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research, 8, e52164. https://doi.org/10.2196/52164
Vancouver
Hakam HT, Prill R, Korte L, Lovreković B, Ostojić M, Ramadanov N, et al. Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research. 2024;8:e52164. doi:10.2196/52164
BibTeX
@article{Hakam2024,
author = {Hakam, Hassan Tarek and Prill, Robert and Korte, Lisa and Lovreković, Bruno and Ostojić, Marko and Ramadanov, Nikolai and Muehlensiepen, Felix},
title = {Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis},
journal = {JMIR Formative Research},
year = {2024},
volume = {8},
pages = {e52164},
doi = {10.2196/52164}
}
EUVIMED