EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Diagnosis in Bytes: Comparing the Diagnostic Accuracy of Google and ChatGPT 3.5 as an Educational Support Tool

Guilherme R Guimaraes, Ricardo G Figueiredo, Caroline Santos Silva, Vanessa Arata, Jean Carlos Z Contreras iD, Cristiano M Gomes iD, Ricardo B Tiraboschi, José Bessa Junior

International Journal of Environmental Research and Public Health · 2024

DOI: 10.3390/ijerph21050580 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift International Journal of Environmental Research and Public Health, 2024. Verfasst von Guilherme R Guimaraes und 7 weiteren Autor:innen. Beteiligte Einrichtung: Programa de Pós-Graduação em Saúde Coletiva, Universidade Estadual de Feira de Santana (UEFS), Feira de Santana 44.036-900, Brazil. Hauptthemen laut MEDLINE: Search Engine. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Adopting advanced digital technologies as diagnostic support tools in healthcare is an unquestionable trend accelerated by the COVID-19 pandemic. However, their accuracy in suggesting diagnoses remains controversial and needs to be explored. We aimed to evaluate and compare the diagnostic accuracy of two free accessible internet search tools: Google and ChatGPT 3.5.

Methods

To assess the effectiveness of both medical platforms, we conducted evaluations using a sample of 60 clinical cases related to urological pathologies. We organized the urological cases into two distinct categories for our analysis: (i) prevalent conditions, which were compiled using the most common symptoms, as outlined by EAU and UpToDate guidelines, and (ii) unusual disorders, identified through case reports published in the ‘Urology Case Reports’ journal from 2022 to 2023. The outcomes were meticulously classified into three categories to determine the accuracy of each platform: “correct diagnosis”, “likely differential diagnosis”, and “incorrect diagnosis”. A group of experts evaluated the responses blindly and randomly.

Results

For commonly encountered urological conditions, Google’s accuracy was 53.3%, with an additional 23.3% of its results falling within a plausible range of differential diagnoses, and the remaining outcomes were incorrect. ChatGPT 3.5 outperformed Google with an accuracy of 86.6%, provided a likely differential diagnosis in 13.3% of cases, and made no unsuitable diagnosis. In evaluating unusual disorders, Google failed to deliver any correct diagnoses but proposed a likely differential diagnosis in 20% of cases. ChatGPT 3.5 identified the proper diagnosis in 16.6% of rare cases and offered a reasonable differential diagnosis in half of the cases.

Conclusion

ChatGPT 3.5 demonstrated higher diagnostic accuracy than Google in both contexts. The platform showed satisfactory accuracy when diagnosing common cases, yet its performance in identifying rare conditions remains limited.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Search EngineHumansCOVID-19InternetUrologic DiseasesSARS-CoV-2Diagnosis, Differential

Stichwörter der Autor:innen

artificial intelligencediagnosismedical educationmedical informatics applicationsurology

Autor:innen und Institutionen

  • Programa de Pós-Graduação em Saúde Coletiva, Universidade Estadual de Feira de Santana (UEFS), Feira de Santana 44.036-900, Brazil.
  • Faculty of Medicine, Universidade de São Paulo (USP), São Paulo 01.246-904, Brazil.

ORCID-Profile: Jean Carlos Z Contreras, Cristiano M Gomes

Zitieren

APA

Guimaraes, G. R., Figueiredo, R. G., Silva, C. S., Arata, V., Contreras, J. C. Z., Gomes, C. M., Tiraboschi, R. B., & Bessa Junior, J. (2024). Diagnosis in Bytes: Comparing the Diagnostic Accuracy of Google and ChatGPT 3.5 as an Educational Support Tool. International Journal of Environmental Research and Public Health. https://doi.org/10.3390/ijerph21050580

Vancouver

Guimaraes GR, Figueiredo RG, Silva CS, Arata V, Contreras JCZ, Gomes CM, et al. Diagnosis in Bytes: Comparing the Diagnostic Accuracy of Google and ChatGPT 3.5 as an Educational Support Tool. International Journal of Environmental Research and Public Health. 2024. doi:10.3390/ijerph21050580

BibTeX
@article{Guimaraes2024,
  author = {Guimaraes, Guilherme R and Figueiredo, Ricardo G and Silva, Caroline Santos and Arata, Vanessa and Contreras, Jean Carlos Z and Gomes, Cristiano M and Tiraboschi, Ricardo B and Bessa Junior, José},
  title = {Diagnosis in Bytes: Comparing the Diagnostic Accuracy of Google and ChatGPT 3.5 as an Educational Support Tool},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2024},
  doi = {10.3390/ijerph21050580}
}
Kostenlos starten →