EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Blinded by the Bot: Benchmarking GPT and Gemini Against Human Authors in Otolaryngology Reviews

Sholem Hack iD, Rebecca Attal, Letizia Nitro, Cecilia Rosso, Anastasia Urbanelli, Antonio Mario Bulfamante, Luigi Angelo Vaira, Omar G Ahmed, Masayoshi Takashima

World Journal of Otorhinolaryngology - Head and Neck Surgery · 2026-10 · 12 (5) · 626-635

DOI: 10.1002/wjo2.70131 ↗

Kurz gefasst

Fachartikel in der Zeitschrift World Journal of Otorhinolaryngology - Head and Neck Surgery, online erschienen am 26.06.2026. Verfasst von Sholem Hack und 8 weiteren Autor:innen. Beteiligte Einrichtung: City St. George's University of London, School of Medicine Program Delivered by University of Nicosia at the Chaim Sheba Medical Center Ramat Gan Isr…. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To compare the quality of scientific review articles generated by two artificial intelligence systems, ChatGPT and Gemini, with those written by human authors in the field of otolaryngology.

Methods

Two otolaryngology topics, chronic rhinosinusitis and infantile subglottic hemangioma, were selected. For each topic, four AI‐generated reviews (GPT‐4.0 and Gemini 2.0; narrative and PRISMA‐style) and one human‐authored peer‐reviewed review were included, yielding a total of 10 manuscripts (8 AI‐generated, 2 human‐authored). A blinded panel of seven board‐certified otolaryngologists evaluated all manuscripts using a 5‐point Likert scale across seven domains: scientific accuracy, depth of content, citation quality, structure and organization, readability and tone, critical insight, and overall scientific quality. Group comparisons were performed using linear mixed‐effects models with random intercepts for reviewer and manuscript. Interrater reliability was assessed using Shrout–Fleiss intraclass correlation coefficients (ICC). Manual verification of AI‐generated references was conducted to assess citation accuracy and fabrication.

Results

Human‐authored manuscripts received the highest ratings across all domains (overall quality 4.50 ± 0.76). GPT‐4.0 demonstrated moderate performance (2.71 ± 1.46 overall), while Gemini 2.0 scored lowest (2.14 ± 1.01). Mixed‐effects modeling demonstrated significant group differences across all domains ( p ≤ 0.008). Citation quality showed one of the largest between‐group differences and strong reliability [ICC(2,1) = 0.68; ICC(2,7) = 0.94]. Manual verification of 123 AI‐generated references revealed high citation accuracy for GPT‐4.0 (89.6% fully accurate; 0% fabricated) compared with Gemini 2.0 (71.7% fully accurate; 23.9% fabricated). Reviewers misclassified 50% of GPT‐4.0 manuscripts as human‐authored, correctly identified 93% of human‐authored manuscripts, and classified 86% of Gemini manuscripts as AI‐generated.

Conclusion

GPT generated fluent, stylistically strong reviews but remained significantly inferior to human‐authored manuscripts in analytical depth and citation integrity. Gemini 2.0 underperformed across all domains and demonstrated a substantial rate of fabricated citations. As large language models become integrated into academic workflows, transparent disclosure, structured fact‐checking, and human oversight remain essential to safeguard scientific reliability.

Schlagwörter

Stichwörter der Autor:innen

GPT‐4.0citation integritygenerative artificial intelligenceotolaryngologypeer reviewscientific writing

Autor:innen und Institutionen

  • City St. George's University of London, School of Medicine Program Delivered by University of Nicosia at the Chaim Sheba Medical Center Ramat Gan Israel.
  • Otolaryngology Unit, Santi Paolo E Carlo Hospital, Department of Health Sciences Università Degli Studi Di Milano Milan Italy.
  • Department of Surgical Sciences, Otorhinolaryngology Unit University of Turin Turin Italy.
  • Otolaryngology Unit, Department of Pediatric Surgery "Vittore Buzzi" Children Hospital Milan Italy.
  • Maxillofacial Surgery Operative Unit, Department of Medicine Surgery and Pharmacy University of Sassari Sassari Italy.
  • Department of Otolaryngology-Head and Neck Surgery Houston Methodist Hospital Houston Texas USA.

ORCID-Profile: Sholem Hack

Zitieren

APA

Hack, S., Attal, R., Nitro, L., Rosso, C., Urbanelli, A., Bulfamante, A. M., Vaira, L. A., Ahmed, O. G., & Takashima, M. (2026). Blinded by the Bot: Benchmarking GPT and Gemini Against Human Authors in Otolaryngology Reviews. World Journal of Otorhinolaryngology - Head and Neck Surgery, 12(5), 626-635. https://doi.org/10.1002/wjo2.70131

Vancouver

Hack S, Attal R, Nitro L, Rosso C, Urbanelli A, Bulfamante AM, et al. Blinded by the Bot: Benchmarking GPT and Gemini Against Human Authors in Otolaryngology Reviews. World Journal of Otorhinolaryngology - Head and Neck Surgery. 2026;12(5):626-635. doi:10.1002/wjo2.70131

BibTeX
@article{Hack2026,
  author = {Hack, Sholem and Attal, Rebecca and Nitro, Letizia and Rosso, Cecilia and Urbanelli, Anastasia and Bulfamante, Antonio Mario and Vaira, Luigi Angelo and Ahmed, Omar G and Takashima, Masayoshi},
  title = {Blinded by the Bot: Benchmarking GPT and Gemini Against Human Authors in Otolaryngology Reviews},
  journal = {World Journal of Otorhinolaryngology - Head and Neck Surgery},
  year = {2026},
  volume = {12},
  number = {5},
  pages = {626--635},
  doi = {10.1002/wjo2.70131}
}
Kostenlos starten →