EUVIMEDEUVIMED entdecken
Crossref

Enhancing text classification with attention matrices based on BERT

Zhiyi Yu, Hong Li, Jialin Feng

Expert Systems · 2024-03 · 41 (3)

DOI: 10.1111/exsy.13512 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Expert Systems, online erschienen am 22.11.2023. Verfasst von Zhiyi Yu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 46 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text classification is a critical task in the field of natural language processing. While pre‐trained language models like BERT have made significant strides in improving performance in this area, the distinctive dependency information that is present in text has not been fully exploited. Besides, BERT mostly captures phrase‐level information in lower layers, which becomes progressively weaker with the increasing depth of layers. To address these limitations, our work focuses on enhancing text classification through the incorporation of Attention Matrices, particularly in the fine‐tuning process of pre‐trained models like BERT. Our approach, named AM‐BERT, leverages learned dependency relationships as external knowledge to enhance the pre‐trained model by generating attention matrices. In addition, we introduce a new learning strategy that enables the model to retain learned phrase‐level structure information. Extensive experiments and detailed analysis on multiple benchmark datasets demonstrate the effectiveness of our approach in text classification tasks. Furthermore, we show that AM‐BERT achieves stable performance improvements also in named entity recognition tasks.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Yu, Z., Li, H., & Feng, J. (2024). Enhancing text classification with attention matrices based on BERT. Expert Systems, 41(3). https://doi.org/10.1111/exsy.13512

Vancouver

Yu Z, Li H, Feng J. Enhancing text classification with attention matrices based on BERT. Expert Systems. 2024;41(3). doi:10.1111/exsy.13512

BibTeX
@article{Yu2024,
  author = {Yu, Zhiyi and Li, Hong and Feng, Jialin},
  title = {Enhancing text classification with attention matrices based on BERT},
  journal = {Expert Systems},
  year = {2024},
  volume = {41},
  number = {3},
  doi = {10.1111/exsy.13512}
}
Kostenlos starten →