EUVIMEDOpdag EUVIMED
EvaluationsstudieOpen access

Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study

Yuxin Zhang, Jie Song, Cheng Bi, Xin Zheng, Zhichuan Xu, Dan Cao, Bairong Shen

Journal of Medical Internet Research · 2026-05-21 · 28 · e88614-e88614

DOI: 10.2196/88614 ↗

At a glance

Evaluationsstudie in Journal of Medical Internet Research, published online on 2026-05-21. Written by Yuxin Zhang and 6 co-authors. Institution: Department of Medical Oncology, Institutes for Systems Genetics, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital,…. Major MEDLINE topics: Large Language Models, Benchmarking, Microsatellite Instability, Neoplasms. Cited 2 times so far (Crossref); 44 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The reliability of general-purpose large language models (LLMs) for complex clinical tasks in specialized domains such as microsatellite instability (MSI) cancers remains critically uncharacterized. The absence of a domain-specific benchmark to evaluate and guide the optimization of their capabilities across diverse clinical tasks poses unevaluated risks to patient safety.

Objective

This study aimed to develop and validate Microsatellite Instability Cancer Benchmark (MSIC-Bench), a novel, two-tiered benchmark for MSI cancer, covering both consensus and frontier knowledge. Using this framework, we aimed to systematically assess LLM performance across various prompting strategies, identify task-specific weaknesses, and reveal effective pathways for performance improvement.

Methods

We developed MSIC-Bench, a 511-question benchmark derived from clinical guidelines and a curated knowledge base. Three state-of-the-art LLMs (GPT-4o, [OpenAI], Gemini 2.5 Pro [Google], and Claude Opus 4 [Anthropic]) were evaluated using 4 prompting strategies, including vanilla, chain-of-thought, reflection of thoughts, and retrieval-augmented generation (RAG), under both multiple-choice and open-ended modalities. Performance was assessed on accuracy, safety (honesty), error composition, and token usage.

Results

LLMs demonstrated a significant “scaffolding effect,” with accuracy dropping substantially in open-ended scenarios. For non-RAG strategies, the primary failure mode was an internal knowledge deficit. The integration of RAG proved to be the most effective intervention. A domain-aligned RAG strategy not only significantly improved accuracy in complex decision-making tasks but also fundamentally shifted the system’s primary bottleneck from knowledge deficits to retrieval failures. In terms of safety, RAG induced a favorable shift from high-risk fabrications to safer refusals, though this introduced a safety-utility trade-off in the form of false refusals. Notably, our hybrid-RAG configuration, which combined both knowledge sources, demonstrated the most robust and generalizable performance across all tasks.

Conclusions

Current LLMs lack the specialized knowledge required for reliable application in MSI oncology. A well-designed RAG architecture is the pivotal intervention to address this gap. However, its success is not automatic; it transforms the nature of system failure, making retrieval precision and knowledge base quality the new critical determinants of performance and safety. Our findings establish a clear directive for developing trustworthy clinical artificial intelligence: focus must shift toward optimizing the retrieval component and curating high-quality, comprehensive knowledge sources. MSIC-Bench provides a robust framework to guide these future efforts.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsBenchmarkingMicrosatellite InstabilityNeoplasmsHumans

Author keywords

LLMbenchmarkcancerlarge language modelmicrosatellite instabilityprompt engineering

Authors and institutions

  • Department of Medical Oncology, Institutes for Systems Genetics, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital, Sichuan University, No 2222 Xinchuan Road, Gaoxin District, Chengdu,…

Cite

APA

Zhang, Y., Song, J., Bi, C., Zheng, X., Xu, Z., Cao, D., & Shen, B. (2026). Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study. Journal of Medical Internet Research, 28, e88614-e88614. https://doi.org/10.2196/88614

Vancouver

Zhang Y, Song J, Bi C, Zheng X, Xu Z, Cao D, et al. Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study. Journal of Medical Internet Research. 2026;28:e88614-e88614. doi:10.2196/88614

BibTeX
@article{Zhang2026,
  author = {Zhang, Yuxin and Song, Jie and Bi, Cheng and Zheng, Xin and Xu, Zhichuan and Cao, Dan and Shen, Bairong},
  title = {Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e88614--e88614},
  doi = {10.2196/88614}
}
Start for free →