EUVIMEDSkopri EUVIMED
Open access · CC BY 4.0

Transcoding unicode characters with AVX‐512 instructions

Robert Clausecker, Daniel Lemire

Software: Practice and Experience · 2023-12 · 53 (12) · 2430-2462

DOI: 10.1002/spe.3261 ↗

At a glance

Journal article in Software: Practice and Experience, published online on 2023-09-12. Written by Robert Clausecker and 1 co-authors. Cited 3 times so far (Crossref); 19 references. The full text is freely available (open access, CC BY 4.0). Funded by: Natural Sciences and Engineering Research Council of Canada.

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Intel includes in its recent processors a powerful set of instructions capable of processing 512‐bit registers with a single instruction (AVX‐512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the most common formats: UTF‐8 and UTF‐16. With our novel algorithms, we are often twice as fast as the previous best solutions. For example, we transcode Chinese text from UTF‐8 to UTF‐16 at more than 5 GiB using fewer than 2 CPU instructions per character. To ensure reproducibility, we make our software freely available as an open‐source library. Our library is part of the popular Node.js JavaScript runtime.

Funding

  • Natural Sciences and Engineering Research Council of Canada

Cite

APA

Clausecker, R., & Lemire, D. (2023). Transcoding unicode characters with AVX‐512 instructions. Software: Practice and Experience, 53(12), 2430-2462. https://doi.org/10.1002/spe.3261

Vancouver

Clausecker R, Lemire D. Transcoding unicode characters with AVX‐512 instructions. Software: Practice and Experience. 2023;53(12):2430-2462. doi:10.1002/spe.3261

BibTeX
@article{Clausecker2023,
  author = {Clausecker, Robert and Lemire, Daniel},
  title = {Transcoding unicode characters with AVX‐512 instructions},
  journal = {Software: Practice and Experience},
  year = {2023},
  volume = {53},
  number = {12},
  pages = {2430--2462},
  doi = {10.1002/spe.3261}
}
Start for free →