Transcoding unicode characters with AVX‐512 instructions
Software: Practice and Experience · 2023-12 · 53 (12) · 2430-2462
DOI: 10.1002/spe.3261 ↗At a glance
Journal article in Software: Practice and Experience, published online on 2023-09-12. Written by Robert Clausecker and 1 co-authors. Cited 3 times so far (Crossref); 19 references. The full text is freely available (open access, CC BY 4.0). Funded by: Natural Sciences and Engineering Research Council of Canada.
Abstract (original language)
Intel includes in its recent processors a powerful set of instructions capable of processing 512‐bit registers with a single instruction (AVX‐512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the most common formats: UTF‐8 and UTF‐16. With our novel algorithms, we are often twice as fast as the previous best solutions. For example, we transcode Chinese text from UTF‐8 to UTF‐16 at more than 5 GiB using fewer than 2 CPU instructions per character. To ensure reproducibility, we make our software freely available as an open‐source library. Our library is part of the popular Node.js JavaScript runtime.
Funding
- Natural Sciences and Engineering Research Council of Canada
Cite
APA
Clausecker, R., & Lemire, D. (2023). Transcoding unicode characters with AVX‐512 instructions. Software: Practice and Experience, 53(12), 2430-2462. https://doi.org/10.1002/spe.3261
Vancouver
Clausecker R, Lemire D. Transcoding unicode characters with AVX‐512 instructions. Software: Practice and Experience. 2023;53(12):2430-2462. doi:10.1002/spe.3261
BibTeX
@article{Clausecker2023,
author = {Clausecker, Robert and Lemire, Daniel},
title = {Transcoding unicode characters with AVX‐512 instructions},
journal = {Software: Practice and Experience},
year = {2023},
volume = {53},
number = {12},
pages = {2430--2462},
doi = {10.1002/spe.3261}
}
EUVIMED