EUVIMEDSkopri EUVIMED
VergleichsstudieOpen access

Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study

Leistung von GPT-4o und Claude in medizinischen Ethik-Szenarien: vergleichende Studie

Karishma R Desai iD, Anna L Gorsky iD, Nicole A Zelenski iD

JMIR Medical Education · 2026-07-23 · 12 · e70199-e70199

DOI: 10.2196/70199 ↗

Auf einen Blick

GPT-4o erreichte in 200 Ethik- und Rechtsfragen 70 % richtige Antworten und lag damit ähnlich wie medizinische Studierende und Orthopädie-Weiterbildungsärzt:innen. Claude erreichte 89 % und schnitt besser ab als GPT-4o und die menschlichen Vergleichsgruppen.

Fragestellung

Wie schneiden GPT-4o und Claude Sonnet 4 bei Ethikfragen im Vergleich zu Medizinstudierenden und Orthopädie-Resident:innen ab?

Methode

200 zufällig ausgewählte Ethik- oder Rechtsszenarien aus Fragebanken für 3. und 4. Studienjahr sowie Orthopädie-Resident:innen; jede Frage 3-mal an beide KI-Modelle gestellt; Vergleich mit menschlichen Testleistungen.

Ergebnisse

GPT-4o: 140/200 richtig (70 %), ähnlich wie der durchschnittliche menschliche Testwert von 71 % (~142/200). Claude: 180/200 richtig (89 %), signifikant besser als GPT-4o (P< .001) und die menschlichen Testteilnehmenden.

Einschränkungen

GPT-4o zeigte Variabilität zwischen identisch formulierten Durchgängen; die Ursache der Leistungsunterschiede konnte nicht eindeutig isoliert werden.

Schlussfolgerung der Autor:innen

Die Autor:innen folgern, dass GPT-4o Ethikfragen mit ähnlicher Genauigkeit wie die Vergleichsgruppen beantworten kann, während Claude besser abschnitt. Vor einem patientennahen Einsatz sei weitere Forschung zur Verringerung von Output-Variabilität nötig.

Kernaussagen

  • GPT-4o: 70 % richtig, ähnlich wie die menschliche Vergleichsgruppe
  • Claude: 89 % richtig und besser als GPT-4o
  • Output-Variabilität wurde bei GPT-4o häufiger beobachtet

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Vergleichsstudie in JMIR Medical Education, published online on 2026-07-23. Written by Karishma R Desai and 2 co-authors. Institution: Division of Hand Surgery, Department of Orthopaedic Surgery, Emory University, 80 Jesse Hill Jr Drive SE, Atlanta, GA, 30303, United States, 1 404-77…. Major MEDLINE topics: Ethics, Medical. Cited 1 times so far (Crossref); 53 references. The full text is freely available in PubMed Central.

Auf einen BlickAstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

The emergence of AI technology has sparked curiosity regarding the capabilities of large language models (LLMs) in the field of medicine. Minimal research exists regarding the proficiency of various AI models in ethics scenarios, specifically in specialty-based scenarios.

Objective

This study aimed to compare the performance of GPT-4o and Claude Sonnet 4 on ethics questions with that of medical students and orthopedic residents.

Methods

A total of 200 ethical or legal scenario questions were randomly selected from question banks targeted for third- and fourth-year medical students (UWorld, AMBOSS) and orthopedic residents (OrthoBullets). Questions at the medical student level were exclusively text-based, while resident-level questions included text-based questions accompanied by images. Each question was entered identically into each AI model 3 separate times. If answers varied between trials, the answer provided most frequently by the model was used as the selected answer.

Results

GPT-4o correctly answered 140 (70%) of 200 questions, which was similar to the average human test taker score of 71% (~142/200 questions). Claude correctly answered 180 (89%) questions, a score greater than that of human test takers and significantly better than GPT-4o ( P< .001). Claude scored significantly higher than GPT-4o in almost all question categories. GPT-4o provided different responses to identically worded trials for 27 (21%) of 130 general questions and 3 (4%) of 70 orthopedic questions ( P= .002), while Claude did not have a significant difference in variability between these 2 groups (general: 16/130, 12% vs orthopedic: 3/70, 4%; P =.06). Neither GPT-4o or Claude showed a significant difference in the rate of most common incorrect response selection between question categories. Claude showed no significant difference in the rate of most common incorrect response selection between question categories.

Conclusions

These results suggest that GPT-4o can potentially answer both general and specialty-specific ethical questions with similar proficiency to sample groups of both medical students and orthopedic residents, while Claude AI performs significantly better than both humans and GPT-4o. Variables such as AI model framework and training data may drive the observed difference in performance, but the exact cause cannot be definitively isolated without intentional testing. Therefore, further research is needed to ensure safety by minimizing output variability before integrating AI as a patient-facing resource.

Subjects

MeSH terms (MEDLINE)

Ethics, MedicalHumansLarge Language ModelsGenerative Artificial IntelligenceEducational MeasurementStudents, Medical

Author keywords

ChatGPTClaudeartificial intelligenceethicshealthlegalmedical educationmedical studentorthopedicsresident

Authors and institutions

  • Division of Hand Surgery, Department of Orthopaedic Surgery, Emory University, 80 Jesse Hill Jr Drive SE, Atlanta, GA, 30303, United States, 1 404-778-1550.

ORCID profiles: Karishma R Desai, Anna L Gorsky, Nicole A Zelenski

Cite

APA

Desai, K. R., Gorsky, A. L., & Zelenski, N. A. (2026). Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study. JMIR Medical Education, 12, e70199-e70199. https://doi.org/10.2196/70199

Vancouver

Desai KR, Gorsky AL, Zelenski NA. Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study. JMIR Medical Education. 2026;12:e70199-e70199. doi:10.2196/70199

BibTeX
@article{Desai2026,
  author = {Desai, Karishma R and Gorsky, Anna L and Zelenski, Nicole A},
  title = {Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e70199--e70199},
  doi = {10.2196/70199}
}
Start for free →