EUVIMEDOpdag EUVIMED
Open access

Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis

Hassan Tarek Hakam iD, Robert Prill iD, Lisa Korte iD, Bruno Lovreković iD, Marko Ostojić iD, Nikolai Ramadanov iD, Felix Muehlensiepen iD

JMIR Formative Research · 2024-02-16 · 8 · e52164

DOI: 10.2196/52164 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2024-02-16. Written by Hassan Tarek Hakam and 6 co-authors. Institution: Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany. Cited 32 times so far (Crossref); 32 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

As large language models (LLMs) are becoming increasingly integrated into different aspects of health care, questions about the implications for medical academic literature have begun to emerge. Key aspects such as authenticity in academic writing are at stake with artificial intelligence (AI) generating highly linguistically accurate and grammatically sound texts.

Objective

The objective of this study is to compare human-written with AI-generated scientific literature in orthopedics and sports medicine.

Methods

Five original abstracts were selected from the PubMed database. These abstracts were subsequently rewritten with the assistance of 2 LLMs with different degrees of proficiency. Subsequently, researchers with varying degrees of expertise and with different areas of specialization were asked to rank the abstracts according to linguistic and methodological parameters. Finally, researchers had to classify the articles as AI generated or human written.

Results

Neither the researchers nor the AI-detection software could successfully identify the AI-generated texts. Furthermore, the criteria previously suggested in the literature did not correlate with whether the researchers deemed a text to be AI generated or whether they judged the article correctly based on these parameters.

Conclusions

The primary finding of this study was that researchers were unable to distinguish between LLM-generated and human-written texts. However, due to the small sample size, it is not possible to generalize the results of this study. As is the case with any tool used in academic research, the potential to cause harm can be mitigated by relying on the transparency and integrity of the researchers. With scientific integrity at stake, further research with a similar study design should be conducted to determine the magnitude of this issue.

Subjects

Author keywords

AILLMartificial intelligencedetectionfeedbacklarge language modelmedical databaseorthopedicorthopedic surgeryorthopedicsqualitative studyresearchscientific integritysports medicinestudy design

Authors and institutions

  • Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany.
  • Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany.
  • Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany.
  • Center of Health Services Research, Faculty of Health Sciences, University Clinic of Brandenburg, Rüdersdorf bei Berlin, Germany.
  • Faculty of Orthopaedics, University Hospital Merkur, Zagreb, Croatia.
  • Departement of Orthopaedics, University Hospital Mostar, Mostar, Bosnia and Herzegovina.

ORCID profiles: Hassan Tarek Hakam, Robert Prill, Lisa Korte, Bruno Lovreković, Marko Ostojić, Nikolai Ramadanov, Felix Muehlensiepen

Cite

APA

Hakam, H. T., Prill, R., Korte, L., Lovreković, B., Ostojić, M., Ramadanov, N., & Muehlensiepen, F. (2024). Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research, 8, e52164. https://doi.org/10.2196/52164

Vancouver

Hakam HT, Prill R, Korte L, Lovreković B, Ostojić M, Ramadanov N, et al. Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Formative Research. 2024;8:e52164. doi:10.2196/52164

BibTeX
@article{Hakam2024,
  author = {Hakam, Hassan Tarek and Prill, Robert and Korte, Lisa and Lovreković, Bruno and Ostojić, Marko and Ramadanov, Nikolai and Muehlensiepen, Felix},
  title = {Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e52164},
  doi = {10.2196/52164}
}
Start for free →