EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Locality-aware pooling enhances protein language model performance across varied applications

Minh Hoang iD, Mona Singh iD

Bioinformatics · 01.07.2025 · 41 (Supplement_1) · i217-i226

DOI: 10.1093/bioinformatics/btaf178 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioinformatics, online erschienen am 15.07.2025. Verfasst von Minh Hoang und 1 weiteren Autor:innen. Beteiligte Einrichtung: Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Proteins, Computational Biology, Sequence Analysis, Protein. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIGMS NIH HHS, NIH HHS, Princeton Laboratory for Artificial Intelligence.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Motivation Protein language models (PLMs) are amongst the most exciting recent advances for characterizing protein sequences, and have enabled a diverse set of applications, including structure determination, functional property prediction, and mutation impact assessment, all from single protein sequences alone. State-of-the-art PLMs leverage transformer architectures originally developed for natural language processing, and are pre-trained on large protein databases to generate contextualized representations of individual amino acids. To harness the power of these PLMs to predict protein-level properties, these per-residue embeddings are typically “pooled” to fixed-size vectors that are further utilized in downstream prediction networks. Common pooling strategies include Cls-Pooling and Avg-Pooling, but neither of these approaches can capture the local substructures and long-range interactions observed in proteins. Results We propose the use of attention pooling, which can naturally capture these important features of proteins. To make the expensive attention operator (quadratic in the length of the input protein) feasible in practice, we introduce bag-of-mer pooling, or BoM-Pooling, a locality-aware hierarchical pooling technique that combines windowed average pooling with attention pooling. We empirically demonstrate that both full attention pooling and BoM-Pooling outperform previous pooling strategies on three important, diverse tasks: (i) predicting the activities of two proteins as they are varied; (ii) detecting remote homologs; and (iii) predicting signaling protein interactions with peptides. Overall, our work highlights the advantages of biologically inspired pooling techniques in protein sequence modeling and is a step toward more effective adaptations of language models in biological settings. Availability and implementation https://github.com/Singh-Lab/bom-pooling.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingProteinsComputational BiologySequence Analysis, ProteinDatabases, ProteinAlgorithms

Substanzen

Proteins

Autor:innen und Institutionen

  • Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540, United States.
  • Department of Computer Science, Princeton University, Princeton, NJ 08540, United States.

ORCID-Profile: Minh Hoang, Mona Singh

Förderung

  • NIGMS NIH HHS
  • NIH HHS
  • Princeton Laboratory for Artificial Intelligence
  • National Institutes of Health

Zitieren

APA

Hoang, M., & Singh, M. (2025). Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics, 41(Supplement_1), i217-i226. https://doi.org/10.1093/bioinformatics/btaf178

Vancouver

Hoang M, Singh M. Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics. 2025;41(Supplement_1):i217-i226. doi:10.1093/bioinformatics/btaf178

BibTeX
@article{Hoang2025,
  author = {Hoang, Minh and Singh, Mona},
  title = {Locality-aware pooling enhances protein language model performance across varied applications},
  journal = {Bioinformatics},
  year = {2025},
  volume = {41},
  number = {Supplement_1},
  pages = {i217--i226},
  doi = {10.1093/bioinformatics/btaf178}
}
Kostenlos starten →