EUVIMEDEUVIMED entdecken
Crossref

Focused Contrastive Loss for Classification With Pre-Trained Language Models

Jiayuan He, Yuan Li, Zenan Zhai, Biaoyan Fang, Camilo Thorne, Christian Druckenbrodt, Saber Akhondi, Karin Verspoor

IEEE Transactions on Knowledge and Data Engineering · 2024-07 · 36 (7) · 3047-3061

DOI: 10.1109/tkde.2023.3327777 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Knowledge and Data Engineering, 2024-07. Verfasst von Jiayuan He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: Australian Research Council Linkage Project, Elsevier.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Contrastive learning, which learns data representations by contrasting similar and dissimilar instances, has achieved great success in various domains including natural language processing (NLP). Recently, it has been demonstrated that incorporating class labels into contrastive learning, i.e., supervised contrastive learning (SCL), can further enhance the quality of the learned data representations. Although several works have shown empirically that incorporating SCL into classification models leads to better performance, the mechanism of how SCL works for classification is less studied. In this paper, we first investigate how SCL facilitates the classifier learning, where we show that the contrastive region, i.e., the data instances involved in each contrasting operation, has a crucial link to the mechanism of SCL. We reveal that the vanilla SCL is suboptimal since its behavior can be altered by variances in class distributions. Based on this finding, we propose a Focused Contrastive Loss (FoCL) for classification. Compared with SCL, FoCL defines a finer contrastive region, focusing on the data instances surrounding decision boundaries. We conduct extensive experiments on three NLP tasks: text classification, named entity recognition, and relation extraction. Experimental results show consistent and significant improvements of FoCL over strong baselines on various benchmark datasets, especially in few-shot scenarios.

Förderung

  • Australian Research Council Linkage Project
  • Elsevier

Zitieren

APA

He, J., Li, Y., Zhai, Z., Fang, B., Thorne, C., Druckenbrodt, C., Akhondi, S., & Verspoor, K. (2024). Focused Contrastive Loss for Classification With Pre-Trained Language Models. IEEE Transactions on Knowledge and Data Engineering, 36(7), 3047-3061. https://doi.org/10.1109/tkde.2023.3327777

Vancouver

He J, Li Y, Zhai Z, Fang B, Thorne C, Druckenbrodt C, et al. Focused Contrastive Loss for Classification With Pre-Trained Language Models. IEEE Transactions on Knowledge and Data Engineering. 2024;36(7):3047-3061. doi:10.1109/tkde.2023.3327777

BibTeX
@article{He2024,
  author = {He, Jiayuan and Li, Yuan and Zhai, Zenan and Fang, Biaoyan and Thorne, Camilo and Druckenbrodt, Christian and Akhondi, Saber and Verspoor, Karin},
  title = {Focused Contrastive Loss for Classification With Pre-Trained Language Models},
  journal = {IEEE Transactions on Knowledge and Data Engineering},
  year = {2024},
  volume = {36},
  number = {7},
  pages = {3047--3061},
  doi = {10.1109/tkde.2023.3327777}
}
Kostenlos starten →