EUVIMEDDécouvrir EUVIMED
VergleichsstudieOpen access

Comparing the Accuracy of ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash in Answering Frequently Asked Questions About Systemic Lupus Erythematosus: Quantitative Study

Alvina Widhani iD, Suzy Maria iD, Aisha Putri Chairani iD, Nabila Yulianingrum Adella Visco iD, Muhammad Faiz Amirullah Nurhadi iD, Lutfi Airlangga Harjoprawito iD, Dwitya Elvira iD, Irwin Tedja iD, Yuniza Yuniza iD, Deasy Fetarayani iD, Sukamto Koesnoe iD, Bramantya Wicaksana iD and 2 more

JMIR Formative Research · 2026-06-08 · 10 · e89156-e89156

DOI: 10.2196/89156 ↗

At a glance

Vergleichsstudie in JMIR Formative Research, published online on 2026-06-08. Written by Alvina Widhani and 13 co-authors. Institution: Division of Allergy and Clinical Immunology, Department of Internal Medicine, Faculty of Medicine, Universitas Indonesia, Diponegoro No. 71 Jakarta, …. Major MEDLINE topics: Lupus Erythematosus, Systemic. 35 references. The full text is freely available in PubMed Central.

RésuméSubjectsRéférences bibliographiquesCiteRelated studies

Abstract (original language)

Background

Systemic lupus erythematosus (SLE) is a complex, fluctuating disease, creating a continuous need for reliable patient information. A prior study concluded that patients with SLE often turn to the internet, including artificial intelligence (AI) chatbots, for information regarding SLE. The rise of AI chatbots as a primary information source presents a critical challenge regarding the accuracy of the information they provide.

Objective

This study aimed to evaluate the performance of the latest generation of AI chatbots (ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash) in answering frequently asked questions about SLE.

Methods

Twenty-two frequently asked questions about SLE in Bahasa Indonesia (the Indonesian language) were posed to each chatbot. Responses were independently and blindly evaluated for accuracy by 5 clinical immunologists using a 4-point Likert scale. Readability was assessed using the Flesch reading ease score formula. Statistical comparisons for accuracy and readability were performed using repeated-measures ANOVA or the Friedman test, followed by the Bonferroni test for pairwise comparisons. The Spearman ρ was used to evaluate correlations among accuracy, readability, and word count.

Results

Gemini 2.5 Flash demonstrated the highest accuracy, with a mean score of 1.25 (SD 0.53), significantly outperforming ChatGPT-4o (mean 1.71, SD 0.61; P

Subjects

MeSH terms (MEDLINE)

Lupus Erythematosus, SystemicHumansIndonesiaSurveys and Questionnaires

Author keywords

AIChatGPTDeepSeekGeminiSLEartificial intelligencechatbotsystemic lupus erythematosus

Authors and institutions

  • Division of Allergy and Clinical Immunology, Department of Internal Medicine, Faculty of Medicine, Universitas Indonesia, Diponegoro No. 71 Jakarta, Jakarta, 10430, Indonesia, 62 622-1390-4546.
  • Department of Internal Medicine, Cipto Mangunkusumo General Hospital, Jakarta, Indonesia.
  • Division of Allergy and Clinical Immunology, Department of Internal Medicine, Faculty of Medicine, Andalas University, Padang, West Sumatra, Indonesia.
  • Division of Rheumatology, Allergy-Immunology, Infectious Diseases, Department of Medicine, Persahabatan Hospital, Jakarta, Indonesia.
  • Division of Allergy and Clinical Immunology, Department of Internal Medicine, Faculty of Medicine, Sriwijaya University, Palembang, South Sumatra, Indonesia.
  • Division of Allergy and Clinical Immunology, Department of Internal Medicine, Faculty of Medicine, Universitas Airlangga, Surabaya, East Java, Indonesia.

ORCID profiles: Alvina Widhani, Suzy Maria, Aisha Putri Chairani, Nabila Yulianingrum Adella Visco, Muhammad Faiz Amirullah Nurhadi, Lutfi Airlangga Harjoprawito, Dwitya Elvira, Irwin Tedja, Yuniza Yuniza, Deasy Fetarayani, Sukamto Koesnoe, Bramantya Wicaksana, Anshari Saifuddin Hasibuan, Evy Yunihastuti

Cite

APA

Widhani, A., Maria, S., Chairani, A. P., Visco, N. Y. A., Nurhadi, M. F. A., Harjoprawito, L. A., Elvira, D., Tedja, I., Yuniza, Y., Fetarayani, D., Koesnoe, S., Wicaksana, B., Hasibuan, A. S., & Yunihastuti, E. (2026). Comparing the Accuracy of ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash in Answering Frequently Asked Questions About Systemic Lupus Erythematosus: Quantitative Study. JMIR Formative Research, 10, e89156-e89156. https://doi.org/10.2196/89156

Vancouver

Widhani A, Maria S, Chairani AP, Visco NYA, Nurhadi MFA, Harjoprawito LA, et al. Comparing the Accuracy of ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash in Answering Frequently Asked Questions About Systemic Lupus Erythematosus: Quantitative Study. JMIR Formative Research. 2026;10:e89156-e89156. doi:10.2196/89156

BibTeX
@article{Widhani2026,
  author = {Widhani, Alvina and Maria, Suzy and Chairani, Aisha Putri and Visco, Nabila Yulianingrum Adella and Nurhadi, Muhammad Faiz Amirullah and Harjoprawito, Lutfi Airlangga and Elvira, Dwitya and Tedja, Irwin and Yuniza, Yuniza and Fetarayani, Deasy and Koesnoe, Sukamto and Wicaksana, Bramantya and Hasibuan, Anshari Saifuddin and Yunihastuti, Evy},
  title = {Comparing the Accuracy of ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash in Answering Frequently Asked Questions About Systemic Lupus Erythematosus: Quantitative Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e89156--e89156},
  doi = {10.2196/89156}
}
Start for free →