Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations
Bewertung von Large-Language-Model-Plattformen für die medizinische Ernährungstherapie bei Autismus-Spektrum-Störung: Qualität, Leitlinienkonformität, Lesbarkeit und Halluzinationen
Journal of Clinical Medicine · 10.11.2026 · 15 (22) · 8770
DOI: 10.3390/jcm15228770 ↗Auf einen Blick
In einer Querschnitts-Benchmarkstudie wurden 4 LLM-Plattformen zu 60 standardisierten ASD-Ernährungsfragen verglichen. ChatGPT Plus schnitt bei Qualität und Leitlinienkonformität am besten ab; Halluzinationen traten bei Copilot und Gemini auf.
Fragestellung
Wie unterscheiden sich 4 öffentlich verfügbare LLM-Plattformen bei Antworten zu Ernährungsfragen bei ASD hinsichtlich Qualität, Leitlinienkonformität, Lesbarkeit und Halluzinationen?
Methode
Querschnittliche Vergleichsstudie; 60 standardisierte Fragen aus 5 MNT-Domänen wurden je 1-mal an ChatGPT, ChatGPT Plus, Microsoft Copilot und Google Gemini gestellt; 240 Antworten. Bewertung durch 4 Expert:innen; Vergleich per GEE.
Ergebnisse
Die Effekte von Plattform, Domäne und Plattform×Domäne waren für QAMAI, GQS und Leitlinienkonformität signifikant (alle omnibus p<0,05). ChatGPT Plus erzielte die höchsten Mittelwerte. Halluzinationen: 0/60 bei ChatGPT und ChatGPT Plus, 21/60 bei Copilot, 38/60 bei Gemini.
Einschränkungen
Nur 1 Antwort pro Frage-Plattform-Kombination; Variabilität durch Wiederholungen nicht gemessen. Die Halluzinationsraten beziehen sich nur auf den untersuchten Antwortsatz und sind nicht verallgemeinerbar.
Schlussfolgerung der Autor:innen
Die Autoren schließen, dass Qualität, Leitlinienkonformität, Lesbarkeit und Halluzinationsbefunde nicht immer übereinstimmen. Die ASD-Ernährungsinformationen der LLM sollten multidimensional bewertet und fachlich validiert werden.
Kernaussagen
- ChatGPT Plus hatte die höchsten QAMAI-, GQS- und Leitlinienwerte.
- Halluzinationen: 0/60 bei ChatGPT und ChatGPT Plus.
- Flesch-Kincaid Grade Level: 14,62 bis 18,87.
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Leitlinie in der Zeitschrift Journal of Clinical Medicine, online erschienen am 10.11.2026. Verfasst von Rüveyda Esra Özkalaycı und 5 weiteren Autor:innen. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Recep Tayyip Erdoğan University Development Foundation.
Abstract (Originalsprache)
Background/Objectives: Children with autism spectrum disorder (ASD) commonly experience feeding- and nutrition-related problems, while families increasingly encounter large language model (LLM)-based tools as sources of health information. Evidence on the reliability of these tools for ASD-specific medical nutrition therapy (MNT) remains limited. The aim of this study is to compare the responses of four publicly available LLM-based platform configurations to nutrition questions related to ASD in terms of quality, guideline compliance, readability, and hallucination.
Methods
In this cross-sectional comparative benchmark study, 60 standardized, unbiased informational questions across five MNT domains (12 per domain) were each submitted once in separate sessions to four LLM-based platform configurations: ChatGPT, ChatGPT Plus, Microsoft Copilot, and Google Gemini, yielding 240 responses. Only one response was evaluated for each question-platform combination, and variability between repeated productions was not measured. Four nutrition and dietetics experts, blinded to platform identity, assessed response quality using Medical Artificial Intelligence Quality Analysis (QAMAI) and Global Quality Score (GQS), and reviewed compliance with NICE guidelines and the American Academy of Pediatrics clinical report, and evaluated readability and binary hallucination classifications. QAMAI, GQS, and guideline-concordance scores were compared using generalized estimating equations with a Gamma distribution and a log link.
Results
Platform, domain (D), and platform × domain effects were significant for QAMAI, GQS, and guideline concordance (all omnibus p < 0.05). ChatGPT Plus had the highest overall mean QAMAI, GQS, and guideline-concordance scores. Gemini showed relatively high quality ratings but the lowest overall concordance with guidelines. In the analyzed response set, no hallucinations were detected in either the ChatGPT or ChatGPT Plus configurations (0/60 each; 0%). Hallucinations were detected in 21/60 (35.0%) of the Copilot responses and in 38/60 (63.3%) of the Gemini responses. Overall platform comparisons of hallucination classifications across the five domains were significant in all five domains after false discovery rate correction; pairwise differences remained significant in D1, D2, and D5, but not in D3 or D4. Mean Flesch–Kincaid Grade Level scores ranged from 14.62 to 18.87 across platforms.
Conclusions
In this standardized single-generation comparison, overall response quality, guideline compliance, readability, and findings on hallucinations did not always coincide. Observed hallucination rates are limited to the response set examined. The observation of zero events in two ChatGPT configurations does not indicate the absence of hallucination risk, nor does it provide a basis for differentiating the configurations or concluding that they are equivalent in this respect. The binary hallucination rates did not reflect error severity and therefore should not be used to compare the clinical risks of the platforms. The findings are limited to a controlled set of questions and cannot be generalized to emotional, directive, or multi-round caregiver interactions. The ASD nutritional information generated by the LLM should be evaluated multidimensionally and subjected to expert validation. Studies involving repeated independent response generation in each question-platform combination are necessary for comparing hallucination tendencies.
Förderung
- Recep Tayyip Erdoğan University Development Foundation
Zitieren
APA
Özkalaycı, R. E., İleri, G. E., Ercan, Ş. N., Serin, Y., Binzat, U., & Arslan, S. (2026). Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations. Journal of Clinical Medicine, 15(22), 8770. https://doi.org/10.3390/jcm15228770
Vancouver
Özkalaycı RE, İleri GE, Ercan ŞN, Serin Y, Binzat U, Arslan S. Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations. Journal of Clinical Medicine. 2026;15(22):8770. doi:10.3390/jcm15228770
BibTeX
@article{zkalayc2026,
author = {Özkalaycı, Rüveyda Esra and İleri, Gözde Ede and Ercan, Şeyma Nur and Serin, Yeliz and Binzat, Uğur and Arslan, Sedat},
title = {Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations},
journal = {Journal of Clinical Medicine},
year = {2026},
volume = {15},
number = {22},
pages = {8770},
doi = {10.3390/jcm15228770}
}
EUVIMED