EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Benchmarking large language models for genomic knowledge with GeneTuring

Xinyi Shang, Xu Liao, Zhicheng Ji, Wenpin Hou

Briefings in Bioinformatics · 31.08.2025 · 26 (5)

DOI: 10.1093/bib/bbaf492 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Briefings in Bioinformatics, online erschienen am 23.09.2025. Verfasst von Xinyi Shang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics, Mailman School of Public Health, Columbia University, 722 West 168th Street, New York, NY 10032, United States. Hauptthemen laut MEDLINE: Genomics, Benchmarking, Software, Programming Languages. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIGMS NIH HHS, NIH HHS, National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) show promise in biomedical research, but their effectiveness for genomic inquiry remains unclear. We developed GeneTuring, a benchmark consisting of 16 genomics tasks with 1600 curated questions, and manually evaluated 48 000 answers from 10 LLM configurations, including GPT-4o (via API, ChatGPT with web access, and a custom Generative Pretrained Transformer (GPT) setup), GPT-3.5, Claude 3.5, Gemini Advanced, GeneGPT (both slim and full), BioGPT, and BioMedLM. A custom GPT-4o configuration integrated with National Center for Biotechnology Information (NCBI) Application Programming Interfaces (APIs), developed in this study as SeqSnap, achieved the best overall performance. GPT-4o with web access and GeneGPT demonstrated complementary strengths. Our findings highlight both the promise and current limitations of LLMs in genomics, and emphasize the value of combining LLMs with domain-specific tools for robust genomic intelligence. GeneTuring offers a key resource for benchmarking and improving LLMs in biomedical research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

GenomicsBenchmarkingSoftwareProgramming LanguagesComputational BiologyHumansLarge Language Models

Stichwörter der Autor:innen

benchmarkgenomicsknowledge baselarge language model

Autor:innen und Institutionen

  • Department of Biostatistics, Mailman School of Public Health, Columbia University, 722 West 168th Street, New York, NY 10032, United States.
  • Department of Biostatistics and Bioinformatics, Duke University School of Medicine, 2424 Erwin Road, Durham, NC 27705, United States.
  • Data Science Institute, Columbia University, 550 West 120th Street, New York, NY 10027, United States.

Förderung

  • NIGMS NIH HHS
  • NIH HHS
  • National Institutes of Health
  • National Institute of General Medical Sciences

Zitieren

APA

Shang, X., Liao, X., Ji, Z., & Hou, W. (2025). Benchmarking large language models for genomic knowledge with GeneTuring. Briefings in Bioinformatics, 26(5). https://doi.org/10.1093/bib/bbaf492

Vancouver

Shang X, Liao X, Ji Z, Hou W. Benchmarking large language models for genomic knowledge with GeneTuring. Briefings in Bioinformatics. 2025;26(5). doi:10.1093/bib/bbaf492

BibTeX
@article{Shang2025,
  author = {Shang, Xinyi and Liao, Xu and Ji, Zhicheng and Hou, Wenpin},
  title = {Benchmarking large language models for genomic knowledge with GeneTuring},
  journal = {Briefings in Bioinformatics},
  year = {2025},
  volume = {26},
  number = {5},
  doi = {10.1093/bib/bbaf492}
}
Kostenlos starten →