Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations
Applied Sciences · 22.09.2024 · 14 (18) · 8532
DOI: 10.3390/app14188532 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 22.09.2024. Verfasst von Wenqin Wu und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea (NRF).
Abstract (Originalsprache)
In general, it is difficult to obtain a huge, labeled dataset for deep learning-based phoneme recognition in singing voices. Studying singing voices also offers inherent challenges, compared to speech, because of the distinct variations in pitch, duration, and intensity. This paper proposes a detouring method to overcome this insufficient dataset, and applies it to the recognition of Korean phonemes in singing voices. The method started with pre-training the HuBERT, a self-supervised speech representation model, on a large-scale English corpus. The model was then adapted to the Korean speech domain with a relatively small-scale Korean corpus, in which the Korean phonemes were interpreted as similar English ones. Finally, the speech-adapted model was again trained with a tiny-scale Korean singing voice corpus for speech–singing adaptation. In the final adaptation, melodic supervision was chosen, which utilizes pitch information to improve the performance. For evaluation, the performance on multi-level error rates based on Word Error Rate (WER) was taken. Using the HuBERT-based transfer learning for adaptation improved the phoneme-level error rate of Korean speech by as much as 31.19%. Again, on singing voices by melodic supervision, it improved the rate by 0.55%. The significant improvement in speech recognition underscores the considerable potential of a model equipped with general human voice representations captured from the English corpus that can improve phoneme recognition on less target speech data. Moreover, the musical variation in singing voices is beneficial for phoneme recognition in singing voices. The proposed method could be applied to the phoneme recognition of other languages that have less speech and singing voice corpora.
Förderung
- National Research Foundation of Korea (NRF)
Zitieren
APA
Wu, W., & Lee, J. (2024). Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations. Applied Sciences, 14(18), 8532. https://doi.org/10.3390/app14188532
Vancouver
Wu W, Lee J. Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations. Applied Sciences. 2024;14(18):8532. doi:10.3390/app14188532
BibTeX
@article{Wu2024,
author = {Wu, Wenqin and Lee, Joonwhoan},
title = {Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {18},
pages = {8532},
doi = {10.3390/app14188532}
}
EUVIMED