EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access · CC BY 4.0

Quality, readability, and clinical-risk signals of default public-interface LLM responses to vascular and perioperative patient questions: a cross-sectional snapshot

Wei Zhong, Yuanyuan Zhang, Yu Huang, Jin Yang, Guoxue Zheng, Qin Li, Gangzhi Li, Qin Liu

Frontiers in Public Health · 10.08.2026 · 14

DOI: 10.3389/fpubh.2026.1905883 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift Frontiers in Public Health, online erschienen am 10.08.2026. Verfasst von Wei Zhong und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Vascular Surgery, Suining Central Hospital, Suining, China. Hauptthemen laut MEDLINE: Large Language Models, Comprehension, Vascular Diseases, Perioperative Care. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Patients increasingly use public large language model chatbot interfaces to seek health information. In vascular disease and perioperative management, default first responses may influence how patients interpret urgent symptoms, antithrombotic medications, procedural choices, and anesthesia-related safety issues.

Methods

This cross-sectional benchmark study evaluated 110 default first responses returned by five publicly accessible LLM chatbot interfaces during a defined access window on May 28–29, 2026, Beijing time (UTC + 8). Interface names were recorded solely as the public-interface display labels visible at the time of access and should not be interpreted as independently verified API-level model identifiers. Each model was queried with 22 guideline-derived English patient-facing questions, yielding 110 responses. Responses were assessed using DISCERN, Ensuring Quality Information for Patients (EQIP), Global Quality Scale (GQS), Journal of the American Medical Association (JAMA) benchmark criteria, used only as a visible metadata/transparency proxy, six readability formulas, an investigator-developed Guideline Concordance Score, and an investigator-developed Potential Clinical-Risk Severity Flag. Differences across the evaluated public-interface response sets were tested using Friedman tests with Holm-adjusted post hoc comparisons.

Results

Interrater agreement was high for established instruments: DISCERN ICC(A,1) = 0.940, EQIP ICC(A,1) = 0.830, GQS weighted κ = 0.829, and JAMA weighted κ = 0.898. Observed public-interface response performance differed significantly for DISCERN, EQIP, and JAMA criteria (all p < 0.001), and for GQS ( p = 0.011). Within this defined late-May 2026 response set, responses returned by the interface displaying the label ‘Grok-4.3’ had the highest observed sample means for DISCERN (58.68), EQIP (81.14), GQS (4.27), and the JAMA visible metadata/transparency proxy score (1.00). No evaluated public-interface response set achieved recommended sixth-grade readability, and no individual response met all six readability thresholds. Responses returned by the interface displaying the label ‘DeepSeek-v4’ had the most favorable observed readability profile within this sampled response set, although the mean FKGL remained 11.53. These observations should not be interpreted as durable model rankings.

Conclusion

In this English-language benchmark of default first responses from five public LLM interfaces accessed through specific logged-in accounts from a Hong Kong, China IP address during a single late-May 2026 window, 107 of 110 responses received the maximum Guideline Concordance Score, and no response met the prespecified criteria for a high Potential Clinical-Risk Severity Flag. Pronounced ceiling and floor effects preclude conclusions about clinical sufficiency or safety. These findings should not be generalized to non-English use, different health-literacy levels, country-specific emergency-care pathways, other regions or account configurations, or later interface states.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsComprehensionVascular DiseasesPerioperative CareCross-Sectional StudiesHumans

Stichwörter der Autor:innen

clinical risklarge language modelspatient educationperioperative managementreadabilityvascular disease

Autor:innen und Institutionen

  • Department of Vascular Surgery, Suining Central Hospital, Suining, China.
  • Department of Anesthesiology, Suining Central Hospital, Suining, China.

Zitieren

APA

Zhong, W., Zhang, Y., Huang, Y., Yang, J., Zheng, G., Li, Q., Li, G., & Liu, Q. (2026). Quality, readability, and clinical-risk signals of default public-interface LLM responses to vascular and perioperative patient questions: a cross-sectional snapshot. Frontiers in Public Health, 14. https://doi.org/10.3389/fpubh.2026.1905883

Vancouver

Zhong W, Zhang Y, Huang Y, Yang J, Zheng G, Li Q, et al. Quality, readability, and clinical-risk signals of default public-interface LLM responses to vascular and perioperative patient questions: a cross-sectional snapshot. Frontiers in Public Health. 2026;14. doi:10.3389/fpubh.2026.1905883

BibTeX
@article{Zhong2026,
  author = {Zhong, Wei and Zhang, Yuanyuan and Huang, Yu and Yang, Jin and Zheng, Guoxue and Li, Qin and Li, Gangzhi and Liu, Qin},
  title = {Quality, readability, and clinical-risk signals of default public-interface LLM responses to vascular and perioperative patient questions: a cross-sectional snapshot},
  journal = {Frontiers in Public Health},
  year = {2026},
  volume = {14},
  doi = {10.3389/fpubh.2026.1905883}
}
Kostenlos starten →