EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

BioCoder: a benchmark for bioinformatics code generation with large language models

Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, Mark B Gerstein iD

Bioinformatics · 28.06.2024 · 40 (Supplement_1) · i266-i276

DOI: 10.1093/bioinformatics/btae230 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioinformatics, online erschienen am 28.06.2024. Verfasst von Xiangru Tang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Yale University, New Haven, CT 06520, United States. Hauptthemen laut MEDLINE: Computational Biology, Software, Programming Languages, Algorithms. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Schmidt Futures.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Summary Pretrained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (i) Successful models accommodate a long prompt (>2600 tokens) with full context, including functional dependencies. (ii) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% versus up to 25%). Availability and implementation All datasets, benchmark, Docker images, and scripts required for testing are available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.github.io/.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Computational BiologySoftwareProgramming LanguagesAlgorithms

Autor:innen und Institutionen

  • Department of Computer Science, Yale University, New Haven, CT 06520, United States.
  • Google Deepmind, Mountain View, CA 94043, United States.
  • Program in Computational Biology & Bioinformatics, Yale University, New Haven, CT 06520, United States.
  • Department of Molecular Biophysics & Biochemistry, Yale University, New Haven, CT 06520, United States.
  • Department of Statistics & Data Science, Yale University, New Haven, CT 06520, United States.
  • Department of Biomedical Informatics & Data Science, Yale University, New Haven, CT 06520, United States.

ORCID-Profile: Mark B Gerstein

Förderung

  • Schmidt Futures

Zitieren

APA

Tang, X., Qian, B., Gao, R., Chen, J., Chen, X., & Gerstein, M. B. (2024). BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40(Supplement_1), i266-i276. https://doi.org/10.1093/bioinformatics/btae230

Vancouver

Tang X, Qian B, Gao R, Chen J, Chen X, Gerstein MB. BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics. 2024;40(Supplement_1):i266-i276. doi:10.1093/bioinformatics/btae230

BibTeX
@article{Tang2024,
  author = {Tang, Xiangru and Qian, Bill and Gao, Rick and Chen, Jiakang and Chen, Xinyun and Gerstein, Mark B},
  title = {BioCoder: a benchmark for bioinformatics code generation with large language models},
  journal = {Bioinformatics},
  year = {2024},
  volume = {40},
  number = {Supplement_1},
  pages = {i266--i276},
  doi = {10.1093/bioinformatics/btae230}
}
Kostenlos starten →