Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare
Machine Learning and Knowledge Extraction · 2025-10-17 · 7 (4) · 121
DOI: 10.3390/make7040121 ↗At a glance
Journal article in Machine Learning and Knowledge Extraction, published online on 2025-10-17. Written by Lovedeep Gondara and 5 co-authors. Cited 10 times so far (Crossref); 15 references. The full text is freely available (open access, CC BY 4.0).
Abstract (original language)
Objectives
To guide language model (LM) selection by comparing finetuning vs. zero-shot use, generic pretraining vs. domain-adjacent vs. further domain-specific pretraining, and bidirectional language models (BiLMs) such as BERT vs. unidirectional LMs (LLMs) for clinical classification. Materials and Methods: We evaluated BiLMs (RoBERTa, PathologyBERT, Gatortron) and LLM (Mistral nemo instruct 12B) on three British Columbia Cancer Registry (BCCR) pathology classification tasks varying in difficulty/data size. We assessed zero-shot vs. finetuned BiLMs, zero-shot LLM, and further BCCR-specific pretraining using macro-average F1 scores.
Results
Finetuned BiLMs outperformed zero-shot BiLMs and zero-shot LLM. The zero-shot LLM outperformed zero-shot BiLMs but was consistently outperformed by finetuned BiLMs. Domain-adjacent BiLMs generally outperformed generic BiLMs after finetuning. Further domain-specific pretraining boosted complex/low-data task performance, with otherwise modest gains.
Conclusions
For specialized classification, finetuning BiLMs is crucial, often surpassing zero-shot LLMs. Domain-adjacent pretrained models are recommended. Further domain-specific pretraining provides significant performance boosts, especially for complex/low-data scenarios. BiLMs remain relevant, offering strong performance/resource balance for targeted clinical tasks.
Cite
APA
Gondara, L., Simkin, J., Sayle, G., Devji, S., Arbour, G., & Ng, R. (2025). Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare. Machine Learning and Knowledge Extraction, 7(4), 121. https://doi.org/10.3390/make7040121
Vancouver
Gondara L, Simkin J, Sayle G, Devji S, Arbour G, Ng R. Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare. Machine Learning and Knowledge Extraction. 2025;7(4):121. doi:10.3390/make7040121
BibTeX
@article{Gondara2025,
author = {Gondara, Lovedeep and Simkin, Jonathan and Sayle, Graham and Devji, Shebnum and Arbour, Gregory and Ng, Raymond},
title = {Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare},
journal = {Machine Learning and Knowledge Extraction},
year = {2025},
volume = {7},
number = {4},
pages = {121},
doi = {10.3390/make7040121}
}
EUVIMED