EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale

César Piñeiro iD, Juan C Pichel iD

GigaScience · 2022

DOI: 10.1093/gigascience/giad062 ↗

Kurz gefasst

Fachartikel in der Zeitschrift GigaScience, 2022. Verfasst von César Piñeiro und 1 weiteren Autor:innen. Beteiligte Einrichtung: CiTIUS, Universidade de Santiago de Compostela, Santiago de Compostela 15782, Spain. Hauptthemen laut MEDLINE: Big Data, Computational Biology. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: MICINN, European Commission, European Regional Development Fund.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

High-throughput sequencing technologies have led to an unprecedented explosion in the amounts of sequencing data available, which are typically stored using FASTA and FASTQ files. We can find in the literature several tools to process and manipulate those type of files with the aim of transforming sequence data into biological knowledge. However, none of them are well fitted for processing efficiently very large files, likely in the order of terabytes in the following years, since they are based on sequential processing. Only some routines of the well-known seqkit tool are partly parallelized. In any case, its scalability is limited to use few threads on a single computing node.

Results

Our approach, BigSeqKit, takes advantage of a high-performance computing–Big Data framework to parallelize and optimize the commands included in seqkit with the aim of speeding up the manipulation of FASTA/FASTQ files. In this way, in most cases, it is from tens to hundreds of times faster than several state-of-the-art tools. At the same time, our toolkit is easy to use and install on any kind of hardware platform (local server or cluster), and its routines can be used as a bioinformatics library or from the command line.

Conclusions

BigSeqKit is a very complete and ultra-fast toolkit to process and manipulate large FASTA and FASTQ files. It is publicly available at https://github.com/citiususc/BigSeqKit.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Big DataComputational BiologyGene LibraryHigh-Throughput Nucleotide SequencingKnowledge

Stichwörter der Autor:innen

Big DataFASTA/FASTQ filesParallelismPerformance

Autor:innen und Institutionen

  • CiTIUS, Universidade de Santiago de Compostela, Santiago de Compostela 15782, Spain.

ORCID-Profile: César Piñeiro, Juan C Pichel

Förderung

  • MICINN
  • European Commission
  • European Regional Development Fund

Zitieren

APA

Piñeiro, C., & Pichel, J. C. (2022). BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale. GigaScience. https://doi.org/10.1093/gigascience/giad062

Vancouver

Piñeiro C, Pichel JC. BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale. GigaScience. 2022. doi:10.1093/gigascience/giad062

BibTeX
@article{Pieiro2022,
  author = {Piñeiro, César and Pichel, Juan C},
  title = {BigSeqKit : a parallel Big Data toolkit to process FASTA and FASTQ files at scale},
  journal = {GigaScience},
  year = {2022},
  doi = {10.1093/gigascience/giad062}
}
Kostenlos starten →