Benchmarking large language models for genomic knowledge with GeneTuring
Briefings in Bioinformatics · 31.08.2025 · 26 (5)
DOI: 10.1093/bib/bbaf492 ↗Kurz gefasst
Fachartikel in der Zeitschrift Briefings in Bioinformatics, online erschienen am 23.09.2025. Verfasst von Xinyi Shang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics, Mailman School of Public Health, Columbia University, 722 West 168th Street, New York, NY 10032, United States. Hauptthemen laut MEDLINE: Genomics, Benchmarking, Software, Programming Languages. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIGMS NIH HHS, NIH HHS, National Institutes of Health.
Abstract (Originalsprache)
Large language models (LLMs) show promise in biomedical research, but their effectiveness for genomic inquiry remains unclear. We developed GeneTuring, a benchmark consisting of 16 genomics tasks with 1600 curated questions, and manually evaluated 48 000 answers from 10 LLM configurations, including GPT-4o (via API, ChatGPT with web access, and a custom Generative Pretrained Transformer (GPT) setup), GPT-3.5, Claude 3.5, Gemini Advanced, GeneGPT (both slim and full), BioGPT, and BioMedLM. A custom GPT-4o configuration integrated with National Center for Biotechnology Information (NCBI) Application Programming Interfaces (APIs), developed in this study as SeqSnap, achieved the best overall performance. GPT-4o with web access and GeneGPT demonstrated complementary strengths. Our findings highlight both the promise and current limitations of LLMs in genomics, and emphasize the value of combining LLMs with domain-specific tools for robust genomic intelligence. GeneTuring offers a key resource for benchmarking and improving LLMs in biomedical research.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biostatistics, Mailman School of Public Health, Columbia University, 722 West 168th Street, New York, NY 10032, United States.
- Department of Biostatistics and Bioinformatics, Duke University School of Medicine, 2424 Erwin Road, Durham, NC 27705, United States.
- Data Science Institute, Columbia University, 550 West 120th Street, New York, NY 10027, United States.
Förderung
- NIGMS NIH HHS
- NIH HHS
- National Institutes of Health
- National Institute of General Medical Sciences
Zitieren
APA
Shang, X., Liao, X., Ji, Z., & Hou, W. (2025). Benchmarking large language models for genomic knowledge with GeneTuring. Briefings in Bioinformatics, 26(5). https://doi.org/10.1093/bib/bbaf492
Vancouver
Shang X, Liao X, Ji Z, Hou W. Benchmarking large language models for genomic knowledge with GeneTuring. Briefings in Bioinformatics. 2025;26(5). doi:10.1093/bib/bbaf492
BibTeX
@article{Shang2025,
author = {Shang, Xinyi and Liao, Xu and Ji, Zhicheng and Hou, Wenpin},
title = {Benchmarking large language models for genomic knowledge with GeneTuring},
journal = {Briefings in Bioinformatics},
year = {2025},
volume = {26},
number = {5},
doi = {10.1093/bib/bbaf492}
}
EUVIMED