Analysis-ready VCF at Biobank scale using Zarr
GigaScience · 06.01.2025 · 14
DOI: 10.1093/gigascience/giaf049 ↗Kurz gefasst
Fachartikel in der Zeitschrift GigaScience, online erschienen am 02.06.2025. Verfasst von Eric Czech und 15 weiteren Autor:innen. Beteiligte Einrichtung: Open Athena AI Foundation, 1245 Broadway, 16th Floor, New York, NY 10001, USA. Hauptthemen laut MEDLINE: Software, Biological Specimen Banks, Genetic Variation. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 217 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Robertson Foundation and NIH, The New Zealand Institute for Plant & Food Research Ltd Kiwifruit Royalty Investment Programme, Gates Foundation.
Abstract (Originalsprache)
Background
Variant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasizes efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. The Biobank-scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed.
Results
Zarr is a format for storing multidimensional data that is widely used across the sciences, and is ideally suited to massively parallel processing. We present the VCF Zarr specification, an encoding of the VCF data model using Zarr, along with fundamental software infrastructure for efficient and reliable conversion at scale. We show how this format is far more efficient than standard VCF-based approaches, and competitive with specialized methods for storing genotype data in terms of compression ratios and single-threaded calculation performance. We present case studies on subsets of 3 large human datasets (Genomics England: $n$=78,195; Our Future Health: $n$=651,050; All of Us: $n$=245,394) along with whole genome datasets for Norway Spruce ($n$=1,063) and SARS-CoV-2 ($n$=4,484,157). We demonstrate the potential for VCF Zarr to enable a new generation of high-performance and cost-effective applications via illustrative examples using cloud computing and GPUs.
Conclusions
Large row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely used, open-source technologies, has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores, while maintaining compatibility with existing file-oriented workflows.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Open Athena AI Foundation, 1245 Broadway, 16th Floor, New York, NY 10001, USA.
- Related Sciences, 1312 17th St PMB 76870, Denver, CO 80202, USA.
- Independent researcher.
- Tom White Consulting Ltd.
- Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, OX3 7LF, UK.
- The New Zealand Institute for Plant & Food Research Ltd, 74 Gerald Street, Lincoln 7608, New Zealand.
ORCID-Profile: Eric Czech, Ben Jeffery, Timothy R Millar, Benjamin Elsworth, Jérémy Guez, Jonny Hancox, Konrad J Karczewski, Alistair Miles, Sam Tallman, Per Unneberg, Rafal Wojdyla, Shadi Zabad, Jeff Hammerbacher, Jerome Kelleher
Förderung
- Robertson Foundation and NIH
- The New Zealand Institute for Plant & Food Research Ltd Kiwifruit Royalty Investment Programme
- Gates Foundation
- Wellcome Trust
- SciLifeLab & Wallenberg Data Driven Life Science Program
- Knut and Alice Wallenberg Foundation
- National Bioinformatics Infrastructure Sweden (NBIS)
- NHGRI NIH HHS
Zitieren
APA
Czech, E., Tyler, W., White, T., Jeffery, B., Millar, T. R., Elsworth, B., Guez, J., Hancox, J., Karczewski, K. J., Miles, A., Tallman, S., Unneberg, P., Wojdyla, R., Zabad, S., Hammerbacher, J., & Kelleher, J. (2025). Analysis-ready VCF at Biobank scale using Zarr. GigaScience, 14. https://doi.org/10.1093/gigascience/giaf049
Vancouver
Czech E, Tyler W, White T, Jeffery B, Millar TR, Elsworth B, et al. Analysis-ready VCF at Biobank scale using Zarr. GigaScience. 2025;14. doi:10.1093/gigascience/giaf049
BibTeX
@article{Czech2025,
author = {Czech, Eric and Tyler, Will and White, Tom and Jeffery, Ben and Millar, Timothy R and Elsworth, Benjamin and Guez, Jérémy and Hancox, Jonny and Karczewski, Konrad J and Miles, Alistair and Tallman, Sam and Unneberg, Per and Wojdyla, Rafal and Zabad, Shadi and Hammerbacher, Jeff and Kelleher, Jerome},
title = {Analysis-ready VCF at Biobank scale using Zarr},
journal = {GigaScience},
year = {2025},
volume = {14},
doi = {10.1093/gigascience/giaf049}
}
EUVIMED