EUVIMEDOpdag EUVIMED
Open access · CC BY-NC-ND 4.0

Transformer technology in molecular science

Jian Jiang, Lu Ke, Long Chen, Bozheng Dou, Yueying Zhu, Jie Liu iD, Bengong Zhang, Tianshou Zhou, Guo-Wei Wei iD

WIREs Computational Molecular Science · 2024-07 · 14 (4)

DOI: 10.1002/wcms.1725 ↗

At a glance

Journal article in WIREs Computational Molecular Science, published online on 2024-08-04. Written by Jian Jiang and 8 co-authors. Institution: Research Center of Nonlinear Science, School of Mathematical and Physical Sciences, Wuhan Textile University, Wuhan, China. Cited 71 times so far (Crossref); 201 references. The full text is freely available (open access, CC BY-NC-ND 4.0). Funded by: NIAID NIH HHS, NIGMS NIH HHS, National Aeronautics and Space Administration.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

A transformer is the foundational architecture behind large language models designed to handle sequential data by using mechanisms of self‐attention to weigh the importance of different elements, enabling efficient processing and understanding of complex patterns. Recently, transformer‐based models have become some of the most popular and powerful deep learning (DL) algorithms in molecular science, owing to their distinctive architectural characteristics and proficiency in handling intricate data. These models leverage the capacity of transformer architectures to capture complex hierarchical dependencies within sequential data. As the applications of transformers in molecular science are very widespread, in this review, we only focus on the technical aspects of transformer technology in molecule domain. Specifically, we will provide an in‐depth investigation into the algorithms of transformer‐based machine learning techniques in molecular science. The models under consideration include generative pre‐trained transformer (GPT), bidirectional and auto‐regressive transformers (BART), bidirectional encoder representations from transformers (BERT), graph transformer, transformer‐XL, text‐to‐text transfer transformer, vision transformers (ViT), detection transformer (DETR), conformer, contrastive language‐image pre‐training (CLIP), sparse transformers, and mobile and efficient transformers. By examining the inner workings of these models, we aim to elucidate how their architectural innovations contribute to their effectiveness in processing complex molecular data. We will also discuss promising trends in transformer models within the context of molecular science, emphasizing their technical capabilities and potential for interdisciplinary research. This review seeks to provide a comprehensive understanding of the transformer‐based machine learning techniques that are driving advancements in molecular science. This article is categorized under: Data Science > Chemoinformatics Data Science > Artificial Intelligence/Machine Learning

Subjects

Author keywords

Data Science > Artificial Intelligence/Machine LearningData Science > Chemoinformaticsbiologychemistrymachine learningmolecular sciencetransformer technology

Authors and institutions

  • Research Center of Nonlinear Science, School of Mathematical and Physical Sciences, Wuhan Textile University, Wuhan, China.
  • Department of Mathematics, Michigan State University, East Lansing, Michigan, USA.
  • Key Laboratory of Computational Mathematics, Guangdong Province, and School of Mathematics, Sun Yat-sen University, Guangzhou, China.
  • Department of Electrical and Computer Engineering, Michigan State University, East Lansing, Michigan, USA.
  • Department of Biochemistry and Molecular Biology, Michigan State University, East Lansing, Michigan, USA.

ORCID profiles: Jie Liu, Guo-Wei Wei

Funding

  • NIAID NIH HHS
  • NIGMS NIH HHS
  • National Aeronautics and Space Administration
  • National Science Foundation
  • National Institutes of Health
  • Michigan State University Foundation
  • Pfizer

Cite

APA

Jiang, J., Ke, L., Chen, L., Dou, B., Zhu, Y., Liu, J., Zhang, B., Zhou, T., & Wei, G. W. (2024). Transformer technology in molecular science. WIREs Computational Molecular Science, 14(4). https://doi.org/10.1002/wcms.1725

Vancouver

Jiang J, Ke L, Chen L, Dou B, Zhu Y, Liu J, et al. Transformer technology in molecular science. WIREs Computational Molecular Science. 2024;14(4). doi:10.1002/wcms.1725

BibTeX
@article{Jiang2024,
  author = {Jiang, Jian and Ke, Lu and Chen, Long and Dou, Bozheng and Zhu, Yueying and Liu, Jie and Zhang, Bengong and Zhou, Tianshou and Wei, Guo-Wei},
  title = {Transformer technology in molecular science},
  journal = {WIREs Computational Molecular Science},
  year = {2024},
  volume = {14},
  number = {4},
  doi = {10.1002/wcms.1725}
}
Start for free →