EUVIMEDEUVIMED entdecken
Crossref

Studying the Effects of Knowledge Distillation and the Fragility of Lightweight BERT-based Models

Allistair Nallie Konsil, Stephanie Chua, Jacey Lynn Minoi, Md Mizanur Rahman, Gerraint Gillan, Rafazila Ramli, Rasitasam Safii, Ahmad Sofian Shminan, Jun Choi Lee

Pertanika Journal of Science and Technology · 25.06.2026 · 34 (3)

DOI: 10.47836/pjst.34.3.17 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Pertanika Journal of Science and Technology, online erschienen am 25.06.2026. Verfasst von Allistair Nallie Konsil und 8 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) such as BERT (Bidirectional Encoder Representations from Transformers) have caused strides of progress in the field of natural language processing (NLP), with lightweight variants such as DistilBERT, MobileBERT and TinyBERT being developed to lower the resource requirements to deploy the models in real-world settings. However, while past research has investigated improving non-distilled models on a variety of benchmarks by changing their architecture, there are limited studies that explore how the lightweight variants may perform in the same conditions. This study applies modifications and ensemble techniques on lightweight BERT models in extractive question answering (QA) to address this gap. The experiments were conducted using three datasets: SQuAD, AdversarialQA, and a newly curated Sexual and Reproductive Health QA (SRHQA) dataset consisting of 1000 samples. From the results, it was shown that applying the same modifications that would enhance the base BERT models to the lightweight variants generally caused a 0.14 - 5.20% F1 decrease in performance, with marginal exceptions observed in specific dataset-model combinations. Ensembling, on the other hand, showed improvements across all the datasets, ranging from 2.19 - 19.46% F1 over the BERT baseline. The results of the study highlight the sensitivity of the lightweight models, their trade-offs with efficiency, and that an ensemble is a valid approach to utilising the lightweight models without architectural modifications.

Zitieren

APA

Konsil, A. N., Chua, S., Minoi, J. L., Rahman, M. M., Gillan, G., Ramli, R., Safii, R., Shminan, A. S., & Lee, J. C. (2026). Studying the Effects of Knowledge Distillation and the Fragility of Lightweight BERT-based Models. Pertanika Journal of Science and Technology, 34(3). https://doi.org/10.47836/pjst.34.3.17

Vancouver

Konsil AN, Chua S, Minoi JL, Rahman MM, Gillan G, Ramli R, et al. Studying the Effects of Knowledge Distillation and the Fragility of Lightweight BERT-based Models. Pertanika Journal of Science and Technology. 2026;34(3). doi:10.47836/pjst.34.3.17

BibTeX
@article{Konsil2026,
  author = {Konsil, Allistair Nallie and Chua, Stephanie and Minoi, Jacey Lynn and Rahman, Md Mizanur and Gillan, Gerraint and Ramli, Rafazila and Safii, Rasitasam and Shminan, Ahmad Sofian and Lee, Jun Choi},
  title = {Studying the Effects of Knowledge Distillation and the Fragility of Lightweight BERT-based Models},
  journal = {Pertanika Journal of Science and Technology},
  year = {2026},
  volume = {34},
  number = {3},
  doi = {10.47836/pjst.34.3.17}
}
Kostenlos starten →