EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Using artificial intelligence to generate medical literature for urology patients: a comparison of three different large language models

David Pompili, Yasmina Richa, Patrick Collins, Helen Richards, Derek B Hennessey

World Journal of Urology · 2024

DOI: 10.1007/s00345-024-05146-3 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift World Journal of Urology, 2024. Verfasst von David Pompili und 4 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, University College Cork, Cork, Ireland. Hauptthemen laut MEDLINE: Artificial Intelligence, Urology. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University College Cork.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Large language models (LLMs) are a form of artificial intelligence (AI) that uses deep learning techniques to understand, summarize and generate content. The potential benefits of LLMs in healthcare is predicted to be immense. The objective of this study was to examine the quality of patient information leaflets (PILs) produced by 3 LLMs on urological topics.

Methods

Prompts were created to generate PILs from 3 LLMs: ChatGPT-4, PaLM 2 (Google Bard) and Llama 2 (Meta) across four urology topics (circumcision, nephrectomy, overactive bladder syndrome, and transurethral resection of the prostate). PILs were evaluated using a quality assessment checklist. PIL readability was assessed by the Average Reading Level Consensus Calculator.

Results

PILs generated by PaLM 2 had the highest overall average quality score (3.58), followed by Llama 2 (3.34) and ChatGPT-4 (3.08). PaLM 2 generated PILs were of the highest quality in all topics except TURP and was the only LLM to include images. Medical inaccuracies were present in all generated content including instances of significant error. Readability analysis identified PaLM 2 generated PILs as the simplest (age 14–15 average reading level). Llama 2 PILs were the most difficult (age 16–17 average).

Conclusion

While LLMs can generate PILs that may help reduce healthcare professional workload, generated content requires clinician input for accuracy and inclusion of health literacy aids, such as images. LLM-generated PILs were above the average reading level for adults, necessitating improvement in LLM algorithms and/or prompt design. How satisfied patients are to LLM-generated PILs remains to be evaluated.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceUrologyHumansPatient Education as TopicLanguageUrologic Diseases

Stichwörter der Autor:innen

Artificial intelligence (AI)ChatGPTGoogle bardLarge language model (LLM)Patient educationPatient information leaflet

Autor:innen und Institutionen

  • School of Medicine, University College Cork, Cork, Ireland.
  • Department of Urology, Mercy University Hospital, Cork, Ireland.
  • Department of Clinical Psychology, Mercy University Hospital, Cork, Ireland.
  • School of Medicine, University College Cork, Cork, Ireland. derek.hennessey@gmail.com.
  • Department of Urology, Mercy University Hospital, Cork, Ireland. derek.hennessey@gmail.com.

Förderung

  • University College Cork

Zitieren

APA

Pompili, D., Richa, Y., Collins, P., Richards, H., & Hennessey, D. B. (2024). Using artificial intelligence to generate medical literature for urology patients: a comparison of three different large language models. World Journal of Urology. https://doi.org/10.1007/s00345-024-05146-3

Vancouver

Pompili D, Richa Y, Collins P, Richards H, Hennessey DB. Using artificial intelligence to generate medical literature for urology patients: a comparison of three different large language models. World Journal of Urology. 2024. doi:10.1007/s00345-024-05146-3

BibTeX
@article{Pompili2024,
  author = {Pompili, David and Richa, Yasmina and Collins, Patrick and Richards, Helen and Hennessey, Derek B},
  title = {Using artificial intelligence to generate medical literature for urology patients: a comparison of three different large language models},
  journal = {World Journal of Urology},
  year = {2024},
  doi = {10.1007/s00345-024-05146-3}
}
Kostenlos starten →