Locality-aware pooling enhances protein language model performance across varied applications
Bioinformatics · 01.07.2025 · 41 (Supplement_1) · i217-i226
DOI: 10.1093/bioinformatics/btaf178 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics, online erschienen am 15.07.2025. Verfasst von Minh Hoang und 1 weiteren Autor:innen. Beteiligte Einrichtung: Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Proteins, Computational Biology, Sequence Analysis, Protein. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIGMS NIH HHS, NIH HHS, Princeton Laboratory for Artificial Intelligence.
Abstract (Originalsprache)
Motivation Protein language models (PLMs) are amongst the most exciting recent advances for characterizing protein sequences, and have enabled a diverse set of applications, including structure determination, functional property prediction, and mutation impact assessment, all from single protein sequences alone. State-of-the-art PLMs leverage transformer architectures originally developed for natural language processing, and are pre-trained on large protein databases to generate contextualized representations of individual amino acids. To harness the power of these PLMs to predict protein-level properties, these per-residue embeddings are typically “pooled” to fixed-size vectors that are further utilized in downstream prediction networks. Common pooling strategies include Cls-Pooling and Avg-Pooling, but neither of these approaches can capture the local substructures and long-range interactions observed in proteins. Results We propose the use of attention pooling, which can naturally capture these important features of proteins. To make the expensive attention operator (quadratic in the length of the input protein) feasible in practice, we introduce bag-of-mer pooling, or BoM-Pooling, a locality-aware hierarchical pooling technique that combines windowed average pooling with attention pooling. We empirically demonstrate that both full attention pooling and BoM-Pooling outperform previous pooling strategies on three important, diverse tasks: (i) predicting the activities of two proteins as they are varied; (ii) detecting remote homologs; and (iii) predicting signaling protein interactions with peptides. Overall, our work highlights the advantages of biologically inspired pooling techniques in protein sequence modeling and is a step toward more effective adaptations of language models in biological settings. Availability and implementation https://github.com/Singh-Lab/bom-pooling.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Substanzen
Autor:innen und Institutionen
- Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540, United States.
- Department of Computer Science, Princeton University, Princeton, NJ 08540, United States.
ORCID-Profile: Minh Hoang, Mona Singh
Förderung
- NIGMS NIH HHS
- NIH HHS
- Princeton Laboratory for Artificial Intelligence
- National Institutes of Health
Zitieren
APA
Hoang, M., & Singh, M. (2025). Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics, 41(Supplement_1), i217-i226. https://doi.org/10.1093/bioinformatics/btaf178
Vancouver
Hoang M, Singh M. Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics. 2025;41(Supplement_1):i217-i226. doi:10.1093/bioinformatics/btaf178
BibTeX
@article{Hoang2025,
author = {Hoang, Minh and Singh, Mona},
title = {Locality-aware pooling enhances protein language model performance across varied applications},
journal = {Bioinformatics},
year = {2025},
volume = {41},
number = {Supplement_1},
pages = {i217--i226},
doi = {10.1093/bioinformatics/btaf178}
}
EUVIMED