EUVIMEDObjevte EUVIMED
Randomisierte kontrollierte StudieVergleichsstudieOpen access

Comparison of ChatGPT and Internet Research for Clinical Research and Decision-Making in Occupational Medicine: Randomized Controlled Trial

Vergleich von ChatGPT und Internetrecherche für klinische Forschung und Entscheidungsfindung in der Arbeitsmedizin: randomisierte kontrollierte Studie

Felix A Weuthen iD, Nelly Otte iD, Hanif Krabbe iD, Thomas Kraus iD, Julia Krabbe iD

JMIR Formative Research · 2025-05-20 · 9 · e63857-e63857

DOI: 10.2196/63857 ↗

Auf einen Blick

In 56 Ärzt:innen und Medizinstudierenden schnitt ChatGPT bei gezielter Recherche teils besser ab als die eigene Internetrecherche. Klinische Entscheidungen wurden jedoch häufiger mit eigener Recherche korrekt getroffen.

Fragestellung

Ist ChatGPT in der Arbeitsmedizin als Unterstützung für medizinische Recherche und klinische Entscheidungen geeignet?

Methode

Randomisierte kontrollierte Studie mit 56 Teilnehmenden; 3 Fälle zu berufsbedingten Lungenerkrankungen; Vergleich von ChatGPT-basierter Recherche mit eigener Recherche.

Ergebnisse

ChatGPT-Gruppe n=27. Bei spezifischen Recherchen teils bessere Ergebnisse und höhere Selbsteinschätzung der Fachkenntnisse. Klinische Entscheidungen häufiger korrekt mit eigener Recherche; OR 6,00 (95 %-KI 1,54–23,36).

Schlussfolgerung der Autor:innen

ChatGPT kann für gezielte medizinische Recherche nützlich sein. Klinische Entscheidungen sollten derzeit nur unterstützt, nicht vom Sprachmodell selbst getroffen werden.

Kernaussagen

  • Gezielte Recherche: teils bessere Ergebnisse mit ChatGPT
  • Klinische Entscheidungen: häufiger korrekt mit eigener Recherche
  • Selbsteinschätzung der Fachkenntnisse unter ChatGPT höher

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Randomisierte kontrollierte Studie in JMIR Formative Research, published online on 2025-05-20. Written by Felix A Weuthen and 4 co-authors. Institution: Institute of Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen University, Aachen…. Major MEDLINE topics: Biomedical Research, Clinical Decision-Making, Generative Artificial Intelligence, Internet. Cited 8 times so far (Crossref); 40 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Artificial intelligence is becoming a part of daily life and the medical field. Generative artificial intelligence models, such as GPT-4 and ChatGPT, are experiencing a surge in popularity due to their enhanced performance and reliability. However, the application of these models in specialized domains, such as occupational medicine, remains largely unexplored.

Objective

This study aims to assess the potential suitability of a generative large language model, such as ChatGPT, as a support tool for medical research and even clinical decisions in occupational medicine in Germany.

Methods

In this randomized controlled study, the usability of ChatGPT for medical research and clinical decision-making was investigated using a web application developed for this purpose. Eligibility criteria were being a physician or medical student.

Participants

(N=56) were asked to work on 3 cases of occupational lung diseases and answer case-related questions. They were allocated via coin weighted for proportions of physicians in each group into 2 groups. One group researched the cases using an integrated chat application similar to ChatGPT based on the latest GPT-4-Turbo model, while the other used their usual research methods, such as Google, Amboss, or DocCheck. The primary outcome was case performance based on correct answers, while secondary outcomes included changes in specific question accuracy and self-assessed occupational medicine expertise before and after case processing. Group assignment was not traditionally blinded, as the chat window indicated membership; participants only knew the study examined web-based research, not group specifics.

Results

Participants of the ChatGPT group (n=27) showed better performance in specific research, for example, for potentially hazardous substances or activities (eg, case 1: ChatGPT group 2.5 hazardous substances that cause pleural changes versus 1.8 in a group with own research; P =.01; Cohen r =–0.38), and led to an increase in self-assessment with regard to specialist knowledge (from 3.9 to 3.4 in the ChatGPT group vs from 3.5 to 3.4 in the own research group; German school grades between 1=very good and 6=unsatisfactory; P =.047). However, clinical decisions, for example, whether an occupational disease report should be filed, were more often made correctly as a result of the participant’s own research (n=29; eg, case 1: Should an occupational disease report be filed? Yes for 7 participants in the ChatGPT group vs 14 in their own research group; P =.007; odds ratio 6.00, 95% CI 1.54‐23.36).

Conclusions

ChatGPT can be a useful tool for targeted medical research, even for rather specific questions in occupational medicine regarding occupational diseases. However, clinical decisions should currently only be supported and not made by the large language model. Future systems should be critically assessed, even if the initial results are promising.

Subjects

MeSH terms (MEDLINE)

Biomedical ResearchClinical Decision-MakingGenerative Artificial IntelligenceInternetOccupational MedicineAdultFemaleHumansMaleMiddle AgedGermanyReproducibility of Results

Author keywords

occupational lung diseasesAIChatGPTLLMsalgorithmalgorithmsartificial intelligencedecision-makinginternet researchlarge language modelsmedical studentsmedicineoccupational diseasesoccupational lung diseaseoccupational medicine

Authors and institutions

  • Institute of Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen University, Aachen, Germany.
  • Department of Vascular Surgery, St. Josef Hospital Bochum, Katholisches Klinikum Bochum, Medical Faculty, Ruhr University Bochum, Bochum, Germany.
  • Institute for Prevention and Occupational Medicine of the German Social Accident Insurance, Medical Faculty, Ruhr University Bochum, Bürkle de la Camp Platz 1, Bochum, 44789, Germany, 49 3013001 ext 4110.

ORCID profiles: Felix A Weuthen, Nelly Otte, Hanif Krabbe, Thomas Kraus, Julia Krabbe

Cite

APA

Weuthen, F. A., Otte, N., Krabbe, H., Kraus, T., & Krabbe, J. (2025). Comparison of ChatGPT and Internet Research for Clinical Research and Decision-Making in Occupational Medicine: Randomized Controlled Trial. JMIR Formative Research, 9, e63857-e63857. https://doi.org/10.2196/63857

Vancouver

Weuthen FA, Otte N, Krabbe H, Kraus T, Krabbe J. Comparison of ChatGPT and Internet Research for Clinical Research and Decision-Making in Occupational Medicine: Randomized Controlled Trial. JMIR Formative Research. 2025;9:e63857-e63857. doi:10.2196/63857

BibTeX
@article{Weuthen2025,
  author = {Weuthen, Felix A and Otte, Nelly and Krabbe, Hanif and Kraus, Thomas and Krabbe, Julia},
  title = {Comparison of ChatGPT and Internet Research for Clinical Research and Decision-Making in Occupational Medicine: Randomized Controlled Trial},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e63857--e63857},
  doi = {10.2196/63857}
}
Start for free →