EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access

Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study

Yuxin Zhang, Jie Song, Cheng Bi, Xin Zheng, Zhichuan Xu, Dan Cao, Bairong Shen

Journal of Medical Internet Research · 21.05.2026 · 28 · e88614-e88614

DOI: 10.2196/88614 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 21.05.2026. Verfasst von Yuxin Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Oncology, Institutes for Systems Genetics, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital,…. Hauptthemen laut MEDLINE: Large Language Models, Benchmarking, Microsatellite Instability, Neoplasms. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The reliability of general-purpose large language models (LLMs) for complex clinical tasks in specialized domains such as microsatellite instability (MSI) cancers remains critically uncharacterized. The absence of a domain-specific benchmark to evaluate and guide the optimization of their capabilities across diverse clinical tasks poses unevaluated risks to patient safety.

Objective

This study aimed to develop and validate Microsatellite Instability Cancer Benchmark (MSIC-Bench), a novel, two-tiered benchmark for MSI cancer, covering both consensus and frontier knowledge. Using this framework, we aimed to systematically assess LLM performance across various prompting strategies, identify task-specific weaknesses, and reveal effective pathways for performance improvement.

Methods

We developed MSIC-Bench, a 511-question benchmark derived from clinical guidelines and a curated knowledge base. Three state-of-the-art LLMs (GPT-4o, [OpenAI], Gemini 2.5 Pro [Google], and Claude Opus 4 [Anthropic]) were evaluated using 4 prompting strategies, including vanilla, chain-of-thought, reflection of thoughts, and retrieval-augmented generation (RAG), under both multiple-choice and open-ended modalities. Performance was assessed on accuracy, safety (honesty), error composition, and token usage.

Results

LLMs demonstrated a significant “scaffolding effect,” with accuracy dropping substantially in open-ended scenarios. For non-RAG strategies, the primary failure mode was an internal knowledge deficit. The integration of RAG proved to be the most effective intervention. A domain-aligned RAG strategy not only significantly improved accuracy in complex decision-making tasks but also fundamentally shifted the system’s primary bottleneck from knowledge deficits to retrieval failures. In terms of safety, RAG induced a favorable shift from high-risk fabrications to safer refusals, though this introduced a safety-utility trade-off in the form of false refusals. Notably, our hybrid-RAG configuration, which combined both knowledge sources, demonstrated the most robust and generalizable performance across all tasks.

Conclusions

Current LLMs lack the specialized knowledge required for reliable application in MSI oncology. A well-designed RAG architecture is the pivotal intervention to address this gap. However, its success is not automatic; it transforms the nature of system failure, making retrieval precision and knowledge base quality the new critical determinants of performance and safety. Our findings establish a clear directive for developing trustworthy clinical artificial intelligence: focus must shift toward optimizing the retrieval component and curating high-quality, comprehensive knowledge sources. MSIC-Bench provides a robust framework to guide these future efforts.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsBenchmarkingMicrosatellite InstabilityNeoplasmsHumans

Stichwörter der Autor:innen

LLMbenchmarkcancerlarge language modelmicrosatellite instabilityprompt engineering

Autor:innen und Institutionen

  • Department of Medical Oncology, Institutes for Systems Genetics, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital, Sichuan University, No 2222 Xinchuan Road, Gaoxin District, Chengdu,…

Zitieren

APA

Zhang, Y., Song, J., Bi, C., Zheng, X., Xu, Z., Cao, D., & Shen, B. (2026). Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study. Journal of Medical Internet Research, 28, e88614-e88614. https://doi.org/10.2196/88614

Vancouver

Zhang Y, Song J, Bi C, Zheng X, Xu Z, Cao D, et al. Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study. Journal of Medical Internet Research. 2026;28:e88614-e88614. doi:10.2196/88614

BibTeX
@article{Zhang2026,
  author = {Zhang, Yuxin and Song, Jie and Bi, Cheng and Zheng, Xin and Xu, Zhichuan and Cao, Dan and Shen, Bairong},
  title = {Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e88614--e88614},
  doi = {10.2196/88614}
}
Kostenlos starten →