EUVIMEDAtrasti EUVIMED
Open access · CC BY 4.0

Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach

Sebastian Griewing, Fabian Lechner, Niklas Gremke, Stefan Lukac, Wolfgang Janni, Markus Wallwiener, Uwe Wagner, Martin Hirsch, Sebastian Kuhn

Journal of Cancer Research and Clinical Oncology · 2024-10-09 · 150 (10)

DOI: 10.1007/s00432-024-05964-3 ↗

At a glance

Journal article in Journal of Cancer Research and Clinical Oncology, published online on 2024-10-09. Written by Sebastian Griewing and 8 co-authors. Institution: Institute for Digital Medicine, University Hospital Giessen and Marburg, Philipps-University Marburg, Marburg, Germany. s.griewing@uni-marburg.de. Major MEDLINE topics: Breast Neoplasms. Cited 27 times so far (Crossref); 41 references. The full text is freely available (open access, CC BY 4.0). Funded by: Philipps-Universität Marburg.

SantraukaSubjectsBibliografiniai duomenysCiteRelated studies

Abstract (original language)

Purpose

Large language models (LLM) show potential for decision support in breast cancer care. Their use in clinical care is currently prohibited by lack of control over sources used for decision-making, explainability of the decision-making process and health data security issues. Recent development of Small Language Models (SLM) is discussed to address these challenges. This preclinical proof-of-concept study tailors an open-source SLM to the German breast cancer guideline (BC-SLM) to evaluate initial clinical accuracy and technical functionality in a preclinical simulation.

Methods

A multidisciplinary tumor board (MTB) is used as the gold-standard to assess the initial clinical accuracy in terms of concordance of the BC-SLM with MTB and comparing it to two publicly available LLM, ChatGPT3.5 and 4. The study includes 20 fictional patient profiles and recommendations for 5 treatment modalities, resulting in 100 binary treatment recommendations (recommended or not recommended). Statistical evaluation includes concordance with MTB in % including Cohen’s Kappa statistic (κ). Technical functionality is assessed qualitatively in terms of local hosting, adherence to the guideline and information retrieval.

Results

The overall concordance amounts to 86% for BC-SLM (κ = 0.721, p < 0.001), 90% for ChatGPT4 (κ = 0.820, p < 0.001) and 83% for ChatGPT3.5 (κ = 0.661, p < 0.001). Specific concordance for each treatment modality ranges from 65 to 100% for BC-SLM, 85–100% for ChatGPT4, and 55–95% for ChatGPT3.5. The BC-SLM is locally functional, adheres to the standards of the German breast cancer guideline and provides referenced sections for its decision-making.

Conclusion

The tailored BC-SLM shows initial clinical accuracy and technical functionality, with concordance to the MTB that is comparable to publicly-available LLMs like ChatGPT4 and 3.5. This serves as a proof-of-concept for adapting a SLM to an oncological disease and its guideline to address prevailing issues with LLM by ensuring decision transparency, explainability, source control, and data security, which represents a necessary step towards clinical validation and safe use of language models in clinical oncology.

Subjects

MeSH terms (MEDLINE)

Breast NeoplasmsHumansFemaleProof of Concept StudyDecision Support Systems, ClinicalDecision Support TechniquesPractice Guidelines as Topic

Author keywords

Artificial intelligenceBreast cancerClinical oncologyLarge language modelSmall language model

Authors and institutions

  • Institute for Digital Medicine, University Hospital Giessen and Marburg, Philipps-University Marburg, Marburg, Germany. s.griewing@uni-marburg.de.
  • Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Palo Alto, CA, USA. s.griewing@uni-marburg.de.
  • Marburg Gynecological Cancer Center, Giessen and Marburg University Hospital, Philipps-University Marburg, Marburg, Germany. s.griewing@uni-marburg.de.
  • Commission Digital Medicine, German Society for Gynecology and Obstetrics (DGGG), Berlin, Germany. s.griewing@uni-marburg.de.
  • Institute for Digital Medicine, University Hospital Giessen and Marburg, Philipps-University Marburg, Marburg, Germany.
  • Institute for Artificial Intelligence in Medicine, University Hospital Giessen and Marburg, Philipps-University Marburg, Marburg, Germany.

Funding

  • Philipps-Universität Marburg

Cite

APA

Griewing, S., Lechner, F., Gremke, N., Lukac, S., Janni, W., Wallwiener, M., Wagner, U., Hirsch, M., & Kuhn, S. (2024). Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach. Journal of Cancer Research and Clinical Oncology, 150(10). https://doi.org/10.1007/s00432-024-05964-3

Vancouver

Griewing S, Lechner F, Gremke N, Lukac S, Janni W, Wallwiener M, et al. Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach. Journal of Cancer Research and Clinical Oncology. 2024;150(10). doi:10.1007/s00432-024-05964-3

BibTeX
@article{Griewing2024,
  author = {Griewing, Sebastian and Lechner, Fabian and Gremke, Niklas and Lukac, Stefan and Janni, Wolfgang and Wallwiener, Markus and Wagner, Uwe and Hirsch, Martin and Kuhn, Sebastian},
  title = {Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach},
  journal = {Journal of Cancer Research and Clinical Oncology},
  year = {2024},
  volume = {150},
  number = {10},
  doi = {10.1007/s00432-024-05964-3}
}
Start for free →