Artificial intelligence meets HIV education: Comparing three large language models on accuracy, readability, and reliability
International Journal of STD & AIDS · 2026
DOI: 10.1177/09564624251372369 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift International Journal of STD & AIDS, 2026. Verfasst von Özge Eren Korkmaz und 4 weiteren Autor:innen. Beteiligte Einrichtung: Epidemiology, Izmir Dokuz Eylul University, Izmir, Turkey. Hauptthemen laut MEDLINE: HIV Infections, Comprehension, Patient Education as Topic, Artificial Intelligence. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 39 Literaturangaben.
Abstract (Originalsprache)
Background
This study compares three large language models (LLMs) in answering common HIV questions, given ongoing concerns about their accuracy and reliability in patient education.
Methods
Models answered 63 HIV questions. Accuracy (5-point Likert), readability (Flesch-Kincaid, Gunning Fog, Coleman-Liau), and reliability (DISCERN, EQIP) were assessed.
Results
Claude 3.7 Sonnet showed significantly higher accuracy (4.54 ± 0.44) compared to ChatGPT-4o (4.29 ± 0.49) and Gemini Advanced 2.0 Flash (4.31 ± 0.50) ( p < .001). ChatGPT-4o had lower accuracy in disease definition, follow-up, and transmission routes, while Gemini Advanced 2.0 Flash performed poorly in daily life and treatment-related questions. Readability analyses indicated ChatGPT-4o produced the most accessible content according to Flesch-Kincaid and Coleman-Liau indices, whereas Claude 3.7 Sonnet was most comprehensible by Gunning Fog standards. Gemini Advanced 2.0 Flash consistently generated more complex texts across all readability measures ( p < .001). Regarding reliability, Claude 3.7 Sonnet achieved “good” quality on DISCERN, while others were rated “moderate” ( p = .059). On EQIP, Claude 3.7 Sonnet (median 61.8) and ChatGPT-4o (55.3) were classified as “good quality with minor limitations,” whereas Gemini Advanced 2.0 Flash (41.2) was rated “low quality” ( p = .049).
Conclusions
Claude 3.7 Sonnet is preferable for accuracy and reliability, while ChatGPT-4o offers superior readability. Selecting LLMs for HIV education should consider accuracy, readability, and reliability, emphasizing regular assessment of content quality and cultural sensitivity.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Epidemiology, Izmir Dokuz Eylul University, Izmir, Turkey.
- Infectious Diseases and Clinical Microbiology, Izmir Democracy University Buca Seyfi Demirsoy Training Research Hospital, Izmir, Turkey.
- Infectious Diseases and Clinical Microbiology, Denizli Pamukkale University, Denizli, Turkey.
- Infectious Diseases and Clinical Microbiology, Izmir Katip Celebi University Atatürk Training and Research Hospital, Izmir, Turkey.
- Infectious Diseases and Clinical Microbiology, Izmir Democracy University, Izmir, Turkey.
ORCID-Profile: Burcu Açıkalın Arıkan
Zitieren
APA
Eren Korkmaz, Ö., Açıkalın Arıkan, B., Sayın Kutlu, S., Kaptan Aydoğmuş, F., & Sezak, N. (2026). Artificial intelligence meets HIV education: Comparing three large language models on accuracy, readability, and reliability. International Journal of STD & AIDS. https://doi.org/10.1177/09564624251372369
Vancouver
Eren Korkmaz Ö, Açıkalın Arıkan B, Sayın Kutlu S, Kaptan Aydoğmuş F, Sezak N. Artificial intelligence meets HIV education: Comparing three large language models on accuracy, readability, and reliability. International Journal of STD & AIDS. 2026. doi:10.1177/09564624251372369
BibTeX
@article{ErenKorkmaz2026,
author = {Eren Korkmaz, Özge and Açıkalın Arıkan, Burcu and Sayın Kutlu, Selda and Kaptan Aydoğmuş, Figen and Sezak, Nurbanu},
title = {Artificial intelligence meets HIV education: Comparing three large language models on accuracy, readability, and reliability},
journal = {International Journal of STD & AIDS},
year = {2026},
doi = {10.1177/09564624251372369}
}
EUVIMED