EUVIMEDEUVIMED entdecken
Open Access

Detecting Artificial Intelligence–Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study

Berin Doru iD, Christoph Maier iD, Johanna Sophie Busse iD, Thomas Lücke iD, Judith Schönhoff iD, Elena Enax-Krumova iD, Steffen Hessler iD, Maria Berger iD, Marianne Tokic iD

JMIR Medical Education · 03.03.2025 · 11 · e62779

DOI: 10.2196/62779 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 03.03.2025. Verfasst von Berin Doru und 8 weiteren Autor:innen. Beteiligte Einrichtung: University Hospital of Paediatrics and Adolescent Medicine, St. Josef-Hospital, Ruhr University Bochum, Bochum, Germany. Hauptthemen laut MEDLINE: Artificial Intelligence, Students, Medical, Writing. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models, exemplified by ChatGPT, have reached a level of sophistication that makes distinguishing between human- and artificial intelligence (AI)–generated texts increasingly challenging. This has raised concerns in academia, particularly in medicine, where the accuracy and authenticity of written work are paramount.

Objective

This semirandomized controlled study aims to examine the ability of 2 blinded expert groups with different levels of content familiarity—medical professionals and humanities scholars with expertise in textual analysis—to distinguish between longer scientific texts in German written by medical students and those generated by ChatGPT. Additionally, the study sought to analyze the reasoning behind their identification choices, particularly the role of content familiarity and linguistic features.

Methods

Between May and August 2023, a total of 35 experts (medical: n=22; humanities: n=13) were each presented with 2 pairs of texts on different medical topics. Each pair had similar content and structure: 1 text was written by a medical student, and the other was generated by ChatGPT (version 3.5, March 2023). Experts were asked to identify the AI-generated text and justify their choice. These justifications were analyzed through a multistage, interdisciplinary qualitative analysis to identify relevant textual features. Before unblinding, experts rated each text on 6 characteristics: linguistic fluency and spelling/grammatical accuracy, scientific quality, logical coherence, expression of knowledge limitations, formulation of future research questions, and citation quality. Univariate tests and multivariate logistic regression analyses were used to examine associations between participants’ characteristics, their stated reasons for author identification, and the likelihood of correctly determining a text’s authorship.

Results

Overall, in 48 out of 69 (70%) decision rounds, participants accurately identified the AI-generated texts, with minimal difference between groups (medical: 31/43, 72%; humanities: 17/26, 65%; odds ratio [OR] 1.37, 95% CI 0.5-3.9). While content errors had little impact on identification accuracy, stylistic features—particularly redundancy (OR 6.90, 95% CI 1.01-47.1), repetition (OR 8.05, 95% CI 1.25-51.7), and thread/coherence (OR 6.62, 95% CI 1.25-35.2)—played a crucial role in participants’ decisions to identify a text as AI-generated.

Conclusions

The findings suggest that both medical and humanities experts were able to identify ChatGPT-generated texts in medical contexts, with their decisions largely based on linguistic attributes. The accuracy of identification appears to be independent of experts’ familiarity with the text content. As the decision-making process primarily relies on linguistic attributes—such as stylistic features and text coherence—further quasi-experimental studies using texts from other academic disciplines should be conducted to determine whether instructions based on these features can enhance lecturers’ ability to distinguish between student-authored and AI-generated work.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceStudents, MedicalWritingHumansFemaleGermanyMale

Stichwörter der Autor:innen

AIChatGPTLLMsartificial intelligenceauthorshipchatbotdecision-makingdetectionlarge language modelslinguistic qualitylogical coherencemedical studenttextual analysiswriting style

Autor:innen und Institutionen

  • University Hospital of Paediatrics and Adolescent Medicine, St. Josef-Hospital, Ruhr University Bochum, Bochum, Germany.
  • Departement of German Philology, General and Comparative Literary Studies, Ruhr University Bochum, Bochum, Germany.
  • Department of Neurology, BG University Hospital Bergmannsheil gGmbH Bochum, Ruhr University Bochum, Bochum, Germany.
  • German Department, German Linguistics, Ruhr University Bochum, Bochum, Germany.
  • German Department, Digital Forensic Linguistics, Ruhr University Bochum, Bochum, Germany.
  • Department for Medical Informatics, Biometry and Epidemiology, Ruhr University Bochum, Bochum, Germany.

ORCID-Profile: Berin Doru, Christoph Maier, Johanna Sophie Busse, Thomas Lücke, Judith Schönhoff, Elena Enax-Krumova, Steffen Hessler, Maria Berger, Marianne Tokic

Zitieren

APA

Doru, B., Maier, C., Busse, J. S., Lücke, T., Schönhoff, J., Enax-Krumova, E., Hessler, S., Berger, M., & Tokic, M. (2025). Detecting Artificial Intelligence–Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study. JMIR Medical Education, 11, e62779. https://doi.org/10.2196/62779

Vancouver

Doru B, Maier C, Busse JS, Lücke T, Schönhoff J, Enax-Krumova E, et al. Detecting Artificial Intelligence–Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study. JMIR Medical Education. 2025;11:e62779. doi:10.2196/62779

BibTeX
@article{Doru2025,
  author = {Doru, Berin and Maier, Christoph and Busse, Johanna Sophie and Lücke, Thomas and Schönhoff, Judith and Enax-Krumova, Elena and Hessler, Steffen and Berger, Maria and Tokic, Marianne},
  title = {Detecting Artificial Intelligence–Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e62779},
  doi = {10.2196/62779}
}
Kostenlos starten →