EUVIMEDEUVIMED entdecken
Crossref

Investigating Data Pruning for Pretraining Biological Foundation Models at Scale

Yifan Wu, Jiyue Jiang, Xichen Ye, Yiqi Wang, Chang Zhou, Yitao Xu, Jiayang Chen, He Hu, Weizhong Zhang, Cheng Jin, Jiao Yuan, Yu Li

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (2) · 1294-1302

DOI: 10.1609/aaai.v40i2.37102 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yifan Wu und 11 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Biological foundation models (BioFMs), pretrained on large-scale biological sequences, have recently shown strong potential in providing meaningful representations for diverse downstream bioinformatics tasks. However, such models often rely on millions to billions of training sequences and billions of parameters, resulting in prohibitive computational costs and significant barriers to reproducibility and accessibility—particularly for academic labs. To address these challenges, we investigate the feasibility of data pruning for BioFM pretraining and propose a post-hoc influence-guided data pruning framework tailored to biological domains. Our approach first introduces a subset-based self-influence formulation that enables efficient estimation of sample importance at low computational cost. Built upon this, we propose two simple yet effective selection strategies: Top-k Influence (Top I) and Coverage-Centric Influence (CCI). Then, we empirically validate our method on two representative BioFMs: RNA-FM and ESM-C. For RNA, our framework consistently outperforms random selection baselines under an extreme pruning rate of over 99%, which displays our framework's effectiveness. Furthermore, we demonstrate the generalizability of our framework on protein-related tasks using ESM-C. Specifically, our coreset even outperforms random 10x subsets in both RNA and protein settings, revealing substantial redundancy in biological sequence datasets. These findings underscore the potential of influence-guided data pruning to substantially reduce the computational cost of BioFM pretraining, paving the way for more efficient, accessible, and sustainable biological AI research.

Zitieren

APA

Wu, Y., Jiang, J., Ye, X., Wang, Y., Zhou, C., Xu, Y., Chen, J., Hu, H., Zhang, W., Jin, C., Yuan, J., & Li, Y. (2026). Investigating Data Pruning for Pretraining Biological Foundation Models at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 40(2), 1294-1302. https://doi.org/10.1609/aaai.v40i2.37102

Vancouver

Wu Y, Jiang J, Ye X, Wang Y, Zhou C, Xu Y, et al. Investigating Data Pruning for Pretraining Biological Foundation Models at Scale. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(2):1294-1302. doi:10.1609/aaai.v40i2.37102

BibTeX
@article{Wu2026,
  author = {Wu, Yifan and Jiang, Jiyue and Ye, Xichen and Wang, Yiqi and Zhou, Chang and Xu, Yitao and Chen, Jiayang and Hu, He and Zhang, Weizhong and Jin, Cheng and Yuan, Jiao and Li, Yu},
  title = {Investigating Data Pruning for Pretraining Biological Foundation Models at Scale},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {2},
  pages = {1294--1302},
  doi = {10.1609/aaai.v40i2.37102}
}
Kostenlos starten →