EUVIMEDEUVIMED entdecken
LeitlinieOpen Access · CC BY 4.0

Leveraging Guideline-Based Clinical Decision Support Systems with Large Language Models: A Case Study with Breast Cancer

Solène Delourme, Akram Redjdal, Jacques Bouaud, Brigitte Seroussi

Methods of Information in Medicine · 01.09.2024 · 63 (03/04) · 085-096

DOI: 10.1055/a-2528-4299 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Methods of Information in Medicine, erschienen am 01.09.2024. Verfasst von Solène Delourme und 3 weiteren Autor:innen. Beteiligte Einrichtung: Sorbonne Université, Université Sorbonne Paris Nord, INSERM, LIMICS, Paris, France. Hauptthemen laut MEDLINE: Decision Support Systems, Clinical, Breast Neoplasms, Practice Guidelines as Topic, Language. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Multidisciplinary tumor boards (MTBs) have been established in most countries to allow experts collaboratively determine the best treatment decisions for cancer patients. However, MTBs often face challenges such as case overload, which can compromise MTB decision quality. Clinical decision support systems (CDSSs) have been introduced to assist clinicians in this process. Despite their potential, CDSSs are still underutilized in routine practice. The emergence of large language models (LLMs), such as ChatGPT, offers new opportunities to improve the efficiency and usability of traditional CDSSs.

Objectives

OncoDoc2 is a guideline-based CDSS developed using a documentary approach and applied to breast cancer management. This study aims to evaluate the potential of LLMs, used as question-answering (QA) systems, to improve the usability of OncoDoc2 across different prompt engineering techniques (PETs).

Methods

Data extracted from breast cancer patient summaries (BCPSs), together with questions formulated by OncoDoc2, were used to create prompts for various LLMs, and several PETs were designed and tested. Using a sample of 200 randomized BCPSs, LLMs and PETs were initially compared with regard to their responses to OncoDoc2 questions using classic metrics (accuracy, precision, recall, and F1 score). Best performing LLMs and PETs were further assessed by comparing the therapeutic recommendations generated by OncoDoc2, based on LLM inputs, to those provided by MTB clinicians using OncoDoc2. Finally, the best performing method was validated using a new sample of 30 randomized BCPSs.

Results

The combination of Mistral and OpenChat models under the enhanced Zero-Shot PET showed the best performance as a question-answering system. This approach gets a precision of 60.16%, a recall of 54.18%, an F1 score of 56.59%, and an accuracy of 75.57% on the validation set of 30 BCPSs. However, this approach yielded poor results as a CDSS, with only 16.67% of the recommendations generated by OncoDoc2 based on LLM inputs matching the gold standard.

Conclusion

All the criteria in the OncoDoc2 decision tree are crucial for capturing the uniqueness of each patient. Any deviation from a criterion alters the recommendations generated. Despite achieving a good accuracy rate of 75.57%, LLMs still face challenges in reliably understanding complex medical contexts and be effective as CDSSs.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Decision Support Systems, ClinicalBreast NeoplasmsPractice Guidelines as TopicLanguageHumansFemaleLarge Language Models

Autor:innen und Institutionen

  • Sorbonne Université, Université Sorbonne Paris Nord, INSERM, LIMICS, Paris, France.
  • EPITA, Paris, France.
  • Univ Gustave Eiffel, Aix-Marseille Univ, LBA, Marseille, France.
  • Sorbonne Université, AP-HP, Tenon Hospital, Public Health Department, INSERM, Université Sorbonne Paris Nord, Limics, Paris, France.
  • APREC, Paris, France.

Zitieren

APA

Delourme, S., Redjdal, A., Bouaud, J., & Seroussi, B. (2024). Leveraging Guideline-Based Clinical Decision Support Systems with Large Language Models: A Case Study with Breast Cancer. Methods of Information in Medicine, 63(03/04), 085-096. https://doi.org/10.1055/a-2528-4299

Vancouver

Delourme S, Redjdal A, Bouaud J, Seroussi B. Leveraging Guideline-Based Clinical Decision Support Systems with Large Language Models: A Case Study with Breast Cancer. Methods of Information in Medicine. 2024;63(03/04):085-096. doi:10.1055/a-2528-4299

BibTeX
@article{Delourme2024,
  author = {Delourme, Solène and Redjdal, Akram and Bouaud, Jacques and Seroussi, Brigitte},
  title = {Leveraging Guideline-Based Clinical Decision Support Systems with Large Language Models: A Case Study with Breast Cancer},
  journal = {Methods of Information in Medicine},
  year = {2024},
  volume = {63},
  number = {03/04},
  pages = {085--096},
  doi = {10.1055/a-2528-4299}
}
Kostenlos starten →