BioCoder: a benchmark for bioinformatics code generation with large language models
Bioinformatics · 28.06.2024 · 40 (Supplement_1) · i266-i276
DOI: 10.1093/bioinformatics/btae230 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics, online erschienen am 28.06.2024. Verfasst von Xiangru Tang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Yale University, New Haven, CT 06520, United States. Hauptthemen laut MEDLINE: Computational Biology, Software, Programming Languages, Algorithms. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Schmidt Futures.
Abstract (Originalsprache)
Summary Pretrained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (i) Successful models accommodate a long prompt (>2600 tokens) with full context, including functional dependencies. (ii) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% versus up to 25%). Availability and implementation All datasets, benchmark, Docker images, and scripts required for testing are available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.github.io/.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Computer Science, Yale University, New Haven, CT 06520, United States.
- Google Deepmind, Mountain View, CA 94043, United States.
- Program in Computational Biology & Bioinformatics, Yale University, New Haven, CT 06520, United States.
- Department of Molecular Biophysics & Biochemistry, Yale University, New Haven, CT 06520, United States.
- Department of Statistics & Data Science, Yale University, New Haven, CT 06520, United States.
- Department of Biomedical Informatics & Data Science, Yale University, New Haven, CT 06520, United States.
ORCID-Profile: Mark B Gerstein
Förderung
- Schmidt Futures
Zitieren
APA
Tang, X., Qian, B., Gao, R., Chen, J., Chen, X., & Gerstein, M. B. (2024). BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40(Supplement_1), i266-i276. https://doi.org/10.1093/bioinformatics/btae230
Vancouver
Tang X, Qian B, Gao R, Chen J, Chen X, Gerstein MB. BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics. 2024;40(Supplement_1):i266-i276. doi:10.1093/bioinformatics/btae230
BibTeX
@article{Tang2024,
author = {Tang, Xiangru and Qian, Bill and Gao, Rick and Chen, Jiakang and Chen, Xinyun and Gerstein, Mark B},
title = {BioCoder: a benchmark for bioinformatics code generation with large language models},
journal = {Bioinformatics},
year = {2024},
volume = {40},
number = {Supplement_1},
pages = {i266--i276},
doi = {10.1093/bioinformatics/btae230}
}
EUVIMED