Aggregating residue-level protein language model embeddings with optimal transport
Bioinformatics Advances · 26.12.2024 · 5 (1)
DOI: 10.1093/bioadv/vbaf060 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics Advances, online erschienen am 20.03.2025. Verfasst von Navid NaderiAlizadeh und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27705, United States. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Whitehead Scholarship.
Abstract (Originalsprache)
Motivation Protein language models (PLMs) have emerged as powerful approaches for mapping protein sequences into embeddings suitable for various applications. As protein representation schemes, PLMs generate per-token (i.e. per-residue) representations, resulting in variable-sized outputs based on protein length. This variability poses a challenge for protein-level prediction tasks that require uniform-sized embeddings for consistent analysis across different proteins. Previous work has typically used average pooling to summarize token-level PLM outputs, but it is unclear whether this method effectively prioritizes the relevant information across token-level representations. Results We introduce a novel method utilizing optimal transport to convert variable-length PLM outputs into fixed-length representations. We conceptualize per-token PLM outputs as samples from a probabilistic distribution and employ sliced-Wasserstein distances to map these samples against a reference set, creating a Euclidean embedding in the output space. The resulting embedding is agnostic to the length of the input and represents the entire protein. We demonstrate the superiority of our method over average pooling for several downstream prediction tasks, particularly with constrained PLM sizes, enabling smaller-scale PLMs to match or exceed the performance of average-pooled larger-scale PLMs. Our aggregation scheme is especially effective for longer protein sequences by capturing essential information that might be lost through average pooling. Availability and implementation Our implementation code can be found at https://github.com/navid-naderi/PLM_SWE.
Autor:innen und Institutionen
- Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27705, United States.
- Department of Cell Biology, Duke University, Durham, NC 27705, United States.
ORCID-Profile: Navid NaderiAlizadeh, Rohit Singh
Förderung
- Whitehead Scholarship
Zitieren
APA
NaderiAlizadeh, N., & Singh, R. (2024). Aggregating residue-level protein language model embeddings with optimal transport. Bioinformatics Advances, 5(1). https://doi.org/10.1093/bioadv/vbaf060
Vancouver
NaderiAlizadeh N, Singh R. Aggregating residue-level protein language model embeddings with optimal transport. Bioinformatics Advances. 2024;5(1). doi:10.1093/bioadv/vbaf060
BibTeX
@article{NaderiAlizadeh2024,
author = {NaderiAlizadeh, Navid and Singh, Rohit},
title = {Aggregating residue-level protein language model embeddings with optimal transport},
journal = {Bioinformatics Advances},
year = {2024},
volume = {5},
number = {1},
doi = {10.1093/bioadv/vbaf060}
}
EUVIMED