EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare

Lovedeep Gondara, Jonathan Simkin, Graham Sayle, Shebnum Devji, Gregory Arbour, Raymond Ng

Machine Learning and Knowledge Extraction · 2025-10-17 · 7 (4) · 121

DOI: 10.3390/make7040121 ↗

At a glance

Journal article in Machine Learning and Knowledge Extraction, published online on 2025-10-17. Written by Lovedeep Gondara and 5 co-authors. Cited 10 times so far (Crossref); 15 references. The full text is freely available (open access, CC BY 4.0).

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Objectives

To guide language model (LM) selection by comparing finetuning vs. zero-shot use, generic pretraining vs. domain-adjacent vs. further domain-specific pretraining, and bidirectional language models (BiLMs) such as BERT vs. unidirectional LMs (LLMs) for clinical classification. Materials and Methods: We evaluated BiLMs (RoBERTa, PathologyBERT, Gatortron) and LLM (Mistral nemo instruct 12B) on three British Columbia Cancer Registry (BCCR) pathology classification tasks varying in difficulty/data size. We assessed zero-shot vs. finetuned BiLMs, zero-shot LLM, and further BCCR-specific pretraining using macro-average F1 scores.

Results

Finetuned BiLMs outperformed zero-shot BiLMs and zero-shot LLM. The zero-shot LLM outperformed zero-shot BiLMs but was consistently outperformed by finetuned BiLMs. Domain-adjacent BiLMs generally outperformed generic BiLMs after finetuning. Further domain-specific pretraining boosted complex/low-data task performance, with otherwise modest gains.

Conclusions

For specialized classification, finetuning BiLMs is crucial, often surpassing zero-shot LLMs. Domain-adjacent pretrained models are recommended. Further domain-specific pretraining provides significant performance boosts, especially for complex/low-data scenarios. BiLMs remain relevant, offering strong performance/resource balance for targeted clinical tasks.

Cite

APA

Gondara, L., Simkin, J., Sayle, G., Devji, S., Arbour, G., & Ng, R. (2025). Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare. Machine Learning and Knowledge Extraction, 7(4), 121. https://doi.org/10.3390/make7040121

Vancouver

Gondara L, Simkin J, Sayle G, Devji S, Arbour G, Ng R. Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare. Machine Learning and Knowledge Extraction. 2025;7(4):121. doi:10.3390/make7040121

BibTeX
@article{Gondara2025,
  author = {Gondara, Lovedeep and Simkin, Jonathan and Sayle, Graham and Devji, Shebnum and Arbour, Gregory and Ng, Raymond},
  title = {Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2025},
  volume = {7},
  number = {4},
  pages = {121},
  doi = {10.3390/make7040121}
}
Start for free →