Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 35021-35029
DOI: 10.1609/aaai.v40i41.40807 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Rui-Chen Zheng und 8 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Existing speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This uniform token allocation mismatches the intrinsic structure of speech, where information is distributed unevenly over time. To address this, we propose VARSTok, a VAriable-frame-Rate Speech Tokenizer that adapts token allocation based on local feature similarity. VARSTok introduces two key innovations: (1) a temporal-aware density peak clustering algorithm that adaptively segments speech into variable-length units, and (2) a novel implicit duration coding scheme that embeds both content and temporal span into a single token index, eliminating the need for auxiliary duration predictors. Extensive experiments show that VARSTok significantly outperforms strong fixed-rate baselines. Notably, it achieves superior reconstruction naturalness while using up to 23% fewer tokens than a 40 Hz fixed-frame-rate baseline. VARSTok further yields lower word error rates and improved naturalness in zero-shot text-to-speech synthesis. To the best of our knowledge, this is the first work to demonstrate that a fully dynamic, variable-frame-rate acoustic speech tokenizer can be seamlessly integrated into downstream speech language models.
Zitieren
APA
Zheng, R., Liu, W., Du, H., Zhang, Q., Deng, C., Chen, Q., Wang, W., Ai, Y., & Ling, Z. (2026). Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 35021-35029. https://doi.org/10.1609/aaai.v40i41.40807
Vancouver
Zheng R, Liu W, Du H, Zhang Q, Deng C, Chen Q, et al. Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):35021-35029. doi:10.1609/aaai.v40i41.40807
BibTeX
@article{Zheng2026,
author = {Zheng, Rui-Chen and Liu, Wenrui and Du, Hui-Peng and Zhang, Qinglin and Deng, Chong and Chen, Qian and Wang, Wen and Ai, Yang and Ling, Zhen-Hua},
title = {Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {41},
pages = {35021--35029},
doi = {10.1609/aaai.v40i41.40807}
}
EUVIMED