Investigating Data Pruning for Pretraining Biological Foundation Models at Scale
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (2) · 1294-1302
DOI: 10.1609/aaai.v40i2.37102 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yifan Wu und 11 weiteren Autor:innen.
Abstract (Originalsprache)
Biological foundation models (BioFMs), pretrained on large-scale biological sequences, have recently shown strong potential in providing meaningful representations for diverse downstream bioinformatics tasks. However, such models often rely on millions to billions of training sequences and billions of parameters, resulting in prohibitive computational costs and significant barriers to reproducibility and accessibility—particularly for academic labs. To address these challenges, we investigate the feasibility of data pruning for BioFM pretraining and propose a post-hoc influence-guided data pruning framework tailored to biological domains. Our approach first introduces a subset-based self-influence formulation that enables efficient estimation of sample importance at low computational cost. Built upon this, we propose two simple yet effective selection strategies: Top-k Influence (Top I) and Coverage-Centric Influence (CCI). Then, we empirically validate our method on two representative BioFMs: RNA-FM and ESM-C. For RNA, our framework consistently outperforms random selection baselines under an extreme pruning rate of over 99%, which displays our framework's effectiveness. Furthermore, we demonstrate the generalizability of our framework on protein-related tasks using ESM-C. Specifically, our coreset even outperforms random 10x subsets in both RNA and protein settings, revealing substantial redundancy in biological sequence datasets. These findings underscore the potential of influence-guided data pruning to substantially reduce the computational cost of BioFM pretraining, paving the way for more efficient, accessible, and sustainable biological AI research.
Zitieren
APA
Wu, Y., Jiang, J., Ye, X., Wang, Y., Zhou, C., Xu, Y., Chen, J., Hu, H., Zhang, W., Jin, C., Yuan, J., & Li, Y. (2026). Investigating Data Pruning for Pretraining Biological Foundation Models at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 40(2), 1294-1302. https://doi.org/10.1609/aaai.v40i2.37102
Vancouver
Wu Y, Jiang J, Ye X, Wang Y, Zhou C, Xu Y, et al. Investigating Data Pruning for Pretraining Biological Foundation Models at Scale. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(2):1294-1302. doi:10.1609/aaai.v40i2.37102
BibTeX
@article{Wu2026,
author = {Wu, Yifan and Jiang, Jiyue and Ye, Xichen and Wang, Yiqi and Zhou, Chang and Xu, Yitao and Chen, Jiayang and Hu, He and Zhang, Weizhong and Jin, Cheng and Yuan, Jiao and Li, Yu},
title = {Investigating Data Pruning for Pretraining Biological Foundation Models at Scale},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {2},
pages = {1294--1302},
doi = {10.1609/aaai.v40i2.37102}
}
EUVIMED