Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.07.2024 · 20 (7) · 1-29
DOI: 10.1145/3651310 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 16.05.2024. Verfasst von Xiangming Gu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: Ministry of Education in Singapore.
Abstract (Originalsprache)
Automatic lyric transcription (ALT) refers to transcribing singing voices into lyrics, while automatic music transcription (AMT) refers to transcribing singing voices into note events, i.e., musical MIDI notes. Despite these two tasks having significant potential for practical application, they are still nascent. This is because the transcription of lyrics and note events solely from singing audio is notoriously difficult due to the presence of noise contamination, e.g., musical accompaniment, resulting in a degradation of both the intelligibility of sung lyrics and the recognizability of sung notes. To address this challenge, we propose a general framework for implementing multimodal ALT and AMT systems. Additionally, we curate the first multimodal singing dataset, comprising N20EMv1 and N20EMv2, which encompasses audio recordings and videos of lip movements, together with ground truth for lyrics and note events. For model construction, we propose adapting self-supervised learning models from the speech domain as acoustic encoders and visual encoders to alleviate the scarcity of labeled data. We also introduce a residual cross-attention mechanism to effectively integrate features from the audio and video modalities. Through extensive experiments, we demonstrate that our single-modal systems exhibit state-of-the-art performance on both ALT and AMT tasks. Subsequently, through single-modal experiments, we also explore the individual contributions of each modality to the multimodal system. Finally, we combine these and demonstrate the effectiveness of our proposed multimodal systems, particularly in terms of their noise robustness.
Förderung
- Ministry of Education in Singapore
Zitieren
APA
Gu, X., Ou, L., Zeng, W., Zhang, J., Wong, N., & Wang, Y. (2024). Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(7), 1-29. https://doi.org/10.1145/3651310
Vancouver
Gu X, Ou L, Zeng W, Zhang J, Wong N, Wang Y. Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(7):1-29. doi:10.1145/3651310
BibTeX
@article{Gu2024,
author = {Gu, Xiangming and Ou, Longshen and Zeng, Wei and Zhang, Jianan and Wong, Nicholas and Wang, Ye},
title = {Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {7},
pages = {1--29},
doi = {10.1145/3651310}
}
EUVIMED