Transformer technology in molecular science
WIREs Computational Molecular Science · 2024-07 · 14 (4)
DOI: 10.1002/wcms.1725 ↗At a glance
Journal article in WIREs Computational Molecular Science, published online on 2024-08-04. Written by Jian Jiang and 8 co-authors. Institution: Research Center of Nonlinear Science, School of Mathematical and Physical Sciences, Wuhan Textile University, Wuhan, China. Cited 71 times so far (Crossref); 201 references. The full text is freely available (open access, CC BY-NC-ND 4.0). Funded by: NIAID NIH HHS, NIGMS NIH HHS, National Aeronautics and Space Administration.
Abstract (original language)
A transformer is the foundational architecture behind large language models designed to handle sequential data by using mechanisms of self‐attention to weigh the importance of different elements, enabling efficient processing and understanding of complex patterns. Recently, transformer‐based models have become some of the most popular and powerful deep learning (DL) algorithms in molecular science, owing to their distinctive architectural characteristics and proficiency in handling intricate data. These models leverage the capacity of transformer architectures to capture complex hierarchical dependencies within sequential data. As the applications of transformers in molecular science are very widespread, in this review, we only focus on the technical aspects of transformer technology in molecule domain. Specifically, we will provide an in‐depth investigation into the algorithms of transformer‐based machine learning techniques in molecular science. The models under consideration include generative pre‐trained transformer (GPT), bidirectional and auto‐regressive transformers (BART), bidirectional encoder representations from transformers (BERT), graph transformer, transformer‐XL, text‐to‐text transfer transformer, vision transformers (ViT), detection transformer (DETR), conformer, contrastive language‐image pre‐training (CLIP), sparse transformers, and mobile and efficient transformers. By examining the inner workings of these models, we aim to elucidate how their architectural innovations contribute to their effectiveness in processing complex molecular data. We will also discuss promising trends in transformer models within the context of molecular science, emphasizing their technical capabilities and potential for interdisciplinary research. This review seeks to provide a comprehensive understanding of the transformer‐based machine learning techniques that are driving advancements in molecular science. This article is categorized under: Data Science > Chemoinformatics Data Science > Artificial Intelligence/Machine Learning
Subjects
Author keywords
Authors and institutions
- Research Center of Nonlinear Science, School of Mathematical and Physical Sciences, Wuhan Textile University, Wuhan, China.
- Department of Mathematics, Michigan State University, East Lansing, Michigan, USA.
- Key Laboratory of Computational Mathematics, Guangdong Province, and School of Mathematics, Sun Yat-sen University, Guangzhou, China.
- Department of Electrical and Computer Engineering, Michigan State University, East Lansing, Michigan, USA.
- Department of Biochemistry and Molecular Biology, Michigan State University, East Lansing, Michigan, USA.
ORCID profiles: Jie Liu, Guo-Wei Wei
Funding
- NIAID NIH HHS
- NIGMS NIH HHS
- National Aeronautics and Space Administration
- National Science Foundation
- National Institutes of Health
- Michigan State University Foundation
- Pfizer
Cite
APA
Jiang, J., Ke, L., Chen, L., Dou, B., Zhu, Y., Liu, J., Zhang, B., Zhou, T., & Wei, G. W. (2024). Transformer technology in molecular science. WIREs Computational Molecular Science, 14(4). https://doi.org/10.1002/wcms.1725
Vancouver
Jiang J, Ke L, Chen L, Dou B, Zhu Y, Liu J, et al. Transformer technology in molecular science. WIREs Computational Molecular Science. 2024;14(4). doi:10.1002/wcms.1725
BibTeX
@article{Jiang2024,
author = {Jiang, Jian and Ke, Lu and Chen, Long and Dou, Bozheng and Zhu, Yueying and Liu, Jie and Zhang, Bengong and Zhou, Tianshou and Wei, Guo-Wei},
title = {Transformer technology in molecular science},
journal = {WIREs Computational Molecular Science},
year = {2024},
volume = {14},
number = {4},
doi = {10.1002/wcms.1725}
}
EUVIMED