BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale
GigaScience · 2022
DOI: 10.1093/gigascience/giad062 ↗Kurz gefasst
Fachartikel in der Zeitschrift GigaScience, 2022. Verfasst von César Piñeiro und 1 weiteren Autor:innen. Beteiligte Einrichtung: CiTIUS, Universidade de Santiago de Compostela, Santiago de Compostela 15782, Spain. Hauptthemen laut MEDLINE: Big Data, Computational Biology. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: MICINN, European Commission, European Regional Development Fund.
Abstract (Originalsprache)
Background
High-throughput sequencing technologies have led to an unprecedented explosion in the amounts of sequencing data available, which are typically stored using FASTA and FASTQ files. We can find in the literature several tools to process and manipulate those type of files with the aim of transforming sequence data into biological knowledge. However, none of them are well fitted for processing efficiently very large files, likely in the order of terabytes in the following years, since they are based on sequential processing. Only some routines of the well-known seqkit tool are partly parallelized. In any case, its scalability is limited to use few threads on a single computing node.
Results
Our approach, BigSeqKit, takes advantage of a high-performance computing–Big Data framework to parallelize and optimize the commands included in seqkit with the aim of speeding up the manipulation of FASTA/FASTQ files. In this way, in most cases, it is from tens to hundreds of times faster than several state-of-the-art tools. At the same time, our toolkit is easy to use and install on any kind of hardware platform (local server or cluster), and its routines can be used as a bioinformatics library or from the command line.
Conclusions
BigSeqKit is a very complete and ultra-fast toolkit to process and manipulate large FASTA and FASTQ files. It is publicly available at https://github.com/citiususc/BigSeqKit.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- CiTIUS, Universidade de Santiago de Compostela, Santiago de Compostela 15782, Spain.
ORCID-Profile: César Piñeiro, Juan C Pichel
Förderung
- MICINN
- European Commission
- European Regional Development Fund
Zitieren
APA
Piñeiro, C., & Pichel, J. C. (2022). BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale. GigaScience. https://doi.org/10.1093/gigascience/giad062
Vancouver
Piñeiro C, Pichel JC. BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale. GigaScience. 2022. doi:10.1093/gigascience/giad062
BibTeX
@article{Pieiro2022,
author = {Piñeiro, César and Pichel, Juan C},
title = {BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale},
journal = {GigaScience},
year = {2022},
doi = {10.1093/gigascience/giad062}
}
EUVIMED