Enhancing HuBERT for spontaneous speech with acoustic feature fusion
The Journal of the Acoustical Society of America · 01.10.2025 · 158 (4_Supplement) · A259-A259
DOI: 10.1121/10.0040817 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Journal of the Acoustical Society of America, erschienen am 01.10.2025. Verfasst von Ki Woong Moon und 1 weiteren Autor:innen.
Abstract (Originalsprache)
Spontaneous and semi-spontaneous speech pose challenges for automatic speech recognition (ASR) systems due to pronunciation variation, reduction, and disfluencies. This study aims to explore whether the performance of a pre-trained speech model on this type of speech can be improved with the addition of some acoustic-prosodic information. Using the openSMILE toolkit, 10 low-level acoustic-prosodic features are extracted from the eGeMAPSv02 set, including loudness, F0, jitter, shimmer, and the frequencies and bandwidths of the first three formants (F1, F2, and F3). We fine-tuned the HuBERT model (Hubert-large-ls960-ft) on a subset of the GigaSpeech corpus consisting of YouTube and podcast recordings. To align the acoustic features with HuBERT’s frame resolution, the features are normalized, interpolated and projected through a linear projection layer. The projected features are subsequently fused with HuBERT’s hidden representations using a gated fusion mechanism. Preliminary results show that the feature-fusion model outperforms the baseline HuBERT model, particularly in recognizing spontaneous or semi-spontaneous speech. Moreover, the study aims to identify the most informative features among the 88 available in the eGeMAPSv02 set. By assessing each feature's impact on spontaneous speech recognition, the study also investigates the broader applicability of the proposed approach via publicly available datasets.
Zitieren
APA
Moon, K. W., & Brenner, D. (2025). Enhancing HuBERT for spontaneous speech with acoustic feature fusion. The Journal of the Acoustical Society of America, 158(4_Supplement), A259-A259. https://doi.org/10.1121/10.0040817
Vancouver
Moon KW, Brenner D. Enhancing HuBERT for spontaneous speech with acoustic feature fusion. The Journal of the Acoustical Society of America. 2025;158(4_Supplement):A259-A259. doi:10.1121/10.0040817
BibTeX
@article{Moon2025,
author = {Moon, Ki Woong and Brenner, Daniel},
title = {Enhancing HuBERT for spontaneous speech with acoustic feature fusion},
journal = {The Journal of the Acoustical Society of America},
year = {2025},
volume = {158},
number = {4_Supplement},
pages = {A259--A259},
doi = {10.1121/10.0040817}
}
EUVIMED